riptoken 0.2.1__tar.gz → 0.2.3__tar.gz

{riptoken-0.2.1 → riptoken-0.2.3}/.github/workflows/ci.yml

@@ -61,7 +61,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: ["3.9", "3.11", "3.13"]
+        python-version: ["3.9", "3.11", "3.13", "3.14"]
     steps:
       - uses: actions/checkout@v4
 

{riptoken-0.2.1 → riptoken-0.2.3}/.github/workflows/release.yml

@@ -28,7 +28,7 @@ jobs:
         uses: PyO3/maturin-action@v1
         with:
           target: ${{ matrix.target }}
-          args: --release --out dist --features python --interpreter python3.9 python3.10 python3.11 python3.12 python3.13
+          args: --release --out dist --features python --interpreter python3.9 python3.10 python3.11 python3.12 python3.13 python3.14
           manylinux: auto
       - name: Upload wheels
         uses: actions/upload-artifact@v4
@@ -51,7 +51,7 @@ jobs:
         uses: PyO3/maturin-action@v1
         with:
           target: ${{ matrix.target }}
-          args: --release --out dist --features python --interpreter python3.9 python3.10 python3.11 python3.12 python3.13
+          args: --release --out dist --features python --interpreter python3.9 python3.10 python3.11 python3.12 python3.13 python3.14
       - name: Upload wheels
         uses: actions/upload-artifact@v4
         with:

{riptoken-0.2.1 → riptoken-0.2.3}/CHANGELOG.md

@@ -7,6 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.3] — 2026-04-11
+
+Minor release. Closes the remaining tiktoken API surface gap by wrapping
+the Rust `CoreBPE` in a new Python `Encoding` class that forwards every
+non-hot-path attribute and method to the underlying `tiktoken.Encoding`.
+Code written against `tiktoken.Encoding` — `n_vocab`, `eot_token`,
+`special_tokens_set`, `encode_with_unstable`, `decode_single_token_bytes`,
+etc. — now works unchanged against a `riptoken.get_encoding` result.
+Also adds Python 3.14 to CI and the release wheels matrix.
+
+### Added
+
+- `riptoken.Encoding` — a drop-in replacement for `tiktoken.Encoding`.
+  Hot-path methods (`encode`, `encode_ordinary`, `decode`, `decode_bytes`,
+  `encode_batch`, `encode_ordinary_batch`) execute in riptoken's Rust
+  core and release the GIL; every other attribute falls through to the
+  wrapped `tiktoken.Encoding` via `__getattr__`. `get_encoding` and
+  `encoding_for_model` now return `Encoding` instances instead of bare
+  `CoreBPE` instances.
+- `encode` and `encode_batch` accept `allowed_special="all"` as a
+  sentinel meaning "every special token in the vocabulary", matching
+  tiktoken.
+- Python 3.14 support — added to the CI matrix and to the release
+  workflow's interpreter list for Linux and macOS wheel builds.
+
+### Changed
+
+- `n_vocab` is now a property (int), not a callable, matching tiktoken.
+  Code that called `enc.n_vocab()` needs to drop the parentheses.
+  `CoreBPE.n_vocab()` is unchanged for direct Rust-core users.
+
+## [0.2.2] — 2026-04-11
+
+Patch release. Closes two tiktoken API compatibility gaps in the Python
+bindings so that code written against `tiktoken.Encoding` — including
+the examples in Sebastian Raschka's *Build a Large Language Model
+(From Scratch)* — runs unchanged against a `riptoken.get_encoding`
+instance.
+
+### Added
+
+- `CoreBPE.decode(tokens)` — returns a Python `str`, matching
+  `tiktoken.Encoding.decode`. Invalid UTF-8 sequences (which can occur
+  mid-stream when a multi-byte character spans a token boundary) are
+  replaced with U+FFFD, matching tiktoken's default
+  `errors="replace"` behavior. The existing `decode_bytes` method is
+  unchanged and remains the right choice for strict / streaming
+  decoding.
+
+### Fixed
+
+- `CoreBPE.encode(text)` now works without an explicit
+  `allowed_special` argument, matching tiktoken. Previously
+  `allowed_special` was a required positional parameter, so
+  `tokenizer.encode(raw_text)` raised a `TypeError`. The parameter is
+  now optional and defaults to an empty set (no special tokens
+  recognized), and can still be passed as a positional or keyword
+  argument. `CoreBPE.encode_batch` received the same treatment.
+
 ## [0.2.1] — 2026-04-11
 
 Patch release. Makes `tiktoken` a required runtime dependency so that

{riptoken-0.2.1 → riptoken-0.2.3}/Cargo.lock

@@ -258,7 +258,7 @@ checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
 
 [[package]]
 name = "riptoken"
-version = "0.2.1"
+version = "0.2.3"
 dependencies = [
  "base64",
  "fancy-regex",

{riptoken-0.2.1 → riptoken-0.2.3}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "riptoken"
-version = "0.2.1"
+version = "0.2.3"
 edition = "2024"
 rust-version = "1.85"
 description = "Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken"

{riptoken-0.2.1 → riptoken-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: riptoken
-Version: 0.2.1
+Version: 0.2.3
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
@@ -14,6 +14,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Rust
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Text Processing :: Linguistic

{riptoken-0.2.1 → riptoken-0.2.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "riptoken"
-version = "0.2.1"
+version = "0.2.3"
 description = "Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -21,6 +21,7 @@ classifiers = [
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Programming Language :: Rust",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
     "Topic :: Text Processing :: Linguistic",

riptoken-0.2.3/python/riptoken/__init__.py

@@ -0,0 +1,316 @@
+"""riptoken — fast BPE tokenizer for LLMs.
+
+A drop-in compatible, faster reimplementation of OpenAI's tiktoken. riptoken
+reads the same ``.tiktoken`` vocabulary files and produces byte-identical
+output for every tested corpus.
+
+Quick start
+-----------
+
+The simplest way to get a tokenizer:
+
+>>> import riptoken
+>>> enc = riptoken.get_encoding("o200k_base")
+>>> enc.encode_ordinary("Hello, world!")
+[13225, 11, 2375, 0]
+
+The returned object is a :class:`riptoken.Encoding` — a drop-in replacement
+for :class:`tiktoken.Encoding`. The hot-path methods (``encode``,
+``encode_ordinary``, ``decode``, ``decode_bytes``, and their batch variants)
+run through riptoken's fast Rust core; every other attribute and method
+forwards transparently to the underlying ``tiktoken.Encoding`` instance, so
+code written against the ``tiktoken`` API works unchanged.
+
+Or construct a :class:`CoreBPE` directly from a ``.tiktoken`` file if you
+want to avoid the ``tiktoken`` dependency entirely:
+
+>>> import riptoken
+>>> ranks = riptoken.load_tiktoken_bpe("o200k_base.tiktoken")
+>>> special_tokens = {"<|endoftext|>": 199999}
+>>> pat = r"\\w+|\\s+"  # simplified pattern; use the full o200k pattern in prod
+>>> enc = riptoken.CoreBPE(ranks, special_tokens, pat)
+>>> enc.decode_bytes(enc.encode_ordinary("Hello, world!"))
+b'Hello, world!'
+
+See https://github.com/daechoi/riptoken for full documentation, benchmarks,
+and the canonical ``o200k_base`` regex pattern.
+"""
+
+from __future__ import annotations
+
+import base64
+from pathlib import Path
+from typing import Any, Iterable, Union
+
+from riptoken._riptoken import CoreBPE
+
+__version__ = "0.2.3"
+__all__ = [
+    "CoreBPE",
+    "Encoding",
+    "encoding_for_model",
+    "get_encoding",
+    "load_tiktoken_bpe",
+]
+
+
+def load_tiktoken_bpe(path: Union[str, Path]) -> dict[bytes, int]:
+    """Load a tiktoken ``.tiktoken`` vocabulary file.
+
+    The file format is one token per line::
+
+        <base64-encoded token bytes> <integer rank>
+
+    This is the same format OpenAI ships for ``cl100k_base``, ``o200k_base``,
+    etc. You can obtain vocabularies from the ``tiktoken`` package's cache
+    directory or download them directly from OpenAI's public CDN.
+
+    Parameters
+    ----------
+    path:
+        Filesystem path to the ``.tiktoken`` file.
+
+    Returns
+    -------
+    dict[bytes, int]
+        A dictionary mapping token bytes to their integer rank, suitable for
+        passing to :class:`CoreBPE`'s first positional argument.
+
+    Raises
+    ------
+    FileNotFoundError
+        If ``path`` does not exist.
+    ValueError
+        If a line in the file is malformed.
+    """
+    ranks: dict[bytes, int] = {}
+    with open(path, "rb") as f:
+        for lineno, line in enumerate(f, start=1):
+            if not line.strip():
+                continue
+            parts = line.split()
+            if len(parts) != 2:
+                raise ValueError(
+                    f"{path}: line {lineno}: expected '<b64> <rank>', got {line!r}"
+                )
+            token_b64, rank_str = parts
+            try:
+                token_bytes = base64.b64decode(token_b64)
+                rank = int(rank_str)
+            except (ValueError, base64.binascii.Error) as e:
+                raise ValueError(
+                    f"{path}: line {lineno}: failed to parse token/rank: {e}"
+                ) from e
+            ranks[token_bytes] = rank
+    return ranks
+
+
+class Encoding:
+    """Drop-in replacement for :class:`tiktoken.Encoding`.
+
+    Holds a riptoken :class:`CoreBPE` and the underlying
+    :class:`tiktoken.Encoding` it was built from. Hot-path methods —
+    ``encode``, ``encode_ordinary``, ``decode``, ``decode_bytes``, and their
+    batch variants — execute in riptoken's Rust core and release the GIL.
+    Every other attribute and method is forwarded to the wrapped
+    ``tiktoken.Encoding`` via :meth:`__getattr__`, so properties like
+    ``n_vocab``, ``eot_token``, ``special_tokens_set``, and less-common
+    methods like ``encode_with_unstable`` work transparently.
+
+    You usually obtain an ``Encoding`` via :func:`riptoken.get_encoding` or
+    :func:`riptoken.encoding_for_model`; constructing one directly is
+    unusual.
+
+    Notes
+    -----
+    This class is duck-typed as a ``tiktoken.Encoding`` — it does not
+    subclass it. Code that checks ``isinstance(enc, tiktoken.Encoding)``
+    will need to be updated; code that calls methods on it will not.
+    """
+
+    __slots__ = ("_core", "_tiktoken")
+
+    def __init__(self, core: CoreBPE, tiktoken_encoding: Any) -> None:
+        object.__setattr__(self, "_core", core)
+        object.__setattr__(self, "_tiktoken", tiktoken_encoding)
+
+    # ---- Hot-path methods: run in riptoken's Rust core ----
+
+    def encode_ordinary(self, text: str) -> list[int]:
+        """Encode ``text`` without recognizing any special tokens.
+
+        Equivalent to ``tiktoken.Encoding.encode_ordinary``. Releases the
+        GIL.
+        """
+        return self._core.encode_ordinary(text)
+
+    def encode(
+        self,
+        text: str,
+        allowed_special: Union[set[str], str, None] = None,
+        disallowed_special: Union[set[str], str, None] = None,
+    ) -> list[int]:
+        """Encode ``text``, optionally recognizing special tokens.
+
+        ``allowed_special`` may be a set of special-token strings to
+        recognize, or the sentinel string ``"all"`` to allow every special
+        token in the vocabulary. ``None`` (the default) disallows all.
+
+        ``disallowed_special`` is accepted for ``tiktoken`` signature
+        compatibility but is currently treated as a hint — unrecognized
+        specials pass through as ordinary text rather than raising.
+        """
+        if allowed_special is None:
+            allowed: set[str] = set()
+        elif allowed_special == "all":
+            allowed = set(self._tiktoken.special_tokens_set)
+        else:
+            allowed = set(allowed_special)
+        return self._core.encode(text, allowed)
+
+    def encode_ordinary_batch(self, texts: Iterable[str]) -> list[list[int]]:
+        """Parallel batch version of :meth:`encode_ordinary`.
+
+        Fans out to ``rayon``'s global thread pool, releasing the GIL for
+        the full batch.
+        """
+        return self._core.encode_ordinary_batch(list(texts))
+
+    def encode_batch(
+        self,
+        texts: Iterable[str],
+        allowed_special: Union[set[str], str, None] = None,
+        disallowed_special: Union[set[str], str, None] = None,
+    ) -> list[list[int]]:
+        """Parallel batch version of :meth:`encode`.
+
+        Fans out to ``rayon``'s global thread pool, releasing the GIL for
+        the full batch.
+        """
+        if allowed_special is None:
+            allowed: set[str] = set()
+        elif allowed_special == "all":
+            allowed = set(self._tiktoken.special_tokens_set)
+        else:
+            allowed = set(allowed_special)
+        return self._core.encode_batch(list(texts), allowed)
+
+    def decode(self, tokens: list[int], errors: str = "replace") -> str:
+        """Decode ``tokens`` into a string.
+
+        ``errors`` is accepted for tiktoken signature compatibility; only
+        ``"replace"`` (the default, matching tiktoken) is honored — invalid
+        UTF-8 byte sequences are replaced with U+FFFD.
+        """
+        return self._core.decode(tokens)
+
+    def decode_bytes(self, tokens: list[int]) -> bytes:
+        """Decode ``tokens`` into raw bytes (no UTF-8 conversion)."""
+        return self._core.decode_bytes(tokens)
+
+    # ---- Everything else: forward to tiktoken ----
+
+    def __getattr__(self, name: str) -> Any:
+        # __getattr__ is only called when normal attribute lookup fails, so
+        # the hot-path methods defined above are never routed through here.
+        return getattr(self._tiktoken, name)
+
+    def __repr__(self) -> str:
+        name = getattr(self._tiktoken, "name", "?")
+        return f"<riptoken.Encoding {name!r}>"
+
+
+def _build_encoding(tiktoken_encoding: Any) -> Encoding:
+    """Build a :class:`riptoken.Encoding` from a :class:`tiktoken.Encoding`.
+
+    Reads the vocab, pattern string, and special tokens from stable (if
+    technically private) attributes that tiktoken has exposed since its
+    first release.
+    """
+    # These three attrs have been stable since tiktoken 0.3 and are what
+    # tiktoken itself passes into its own Rust constructor.
+    mergeable_ranks = tiktoken_encoding._mergeable_ranks  # type: ignore[attr-defined]
+    special_tokens = tiktoken_encoding._special_tokens  # type: ignore[attr-defined]
+    pat_str = tiktoken_encoding._pat_str  # type: ignore[attr-defined]
+    core = CoreBPE(mergeable_ranks, special_tokens, pat_str)
+    return Encoding(core, tiktoken_encoding)
+
+
+def get_encoding(name: str) -> Encoding:
+    """Load a named tiktoken encoding (``"gpt2"``, ``"cl100k_base"``, ``"o200k_base"``, ...).
+
+    This is the riptoken equivalent of :func:`tiktoken.get_encoding`. It
+    uses the ``tiktoken`` package to supply the vocabulary file, regex
+    pattern, and special-token map, then wraps them in a
+    :class:`riptoken.Encoding` whose hot-path methods execute in
+    riptoken's faster Rust core. Subsequent calls with the same ``name``
+    reuse tiktoken's in-process cache, and the vocabulary download (if
+    needed) uses tiktoken's on-disk cache at ``~/.cache/tiktoken/`` (or
+    wherever ``TIKTOKEN_CACHE_DIR`` points).
+
+    Parameters
+    ----------
+    name:
+        A tiktoken encoding name, e.g. ``"gpt2"``, ``"r50k_base"``,
+        ``"p50k_base"``, ``"cl100k_base"``, ``"o200k_base"``.
+
+    Returns
+    -------
+    Encoding
+        A drop-in replacement for the corresponding
+        :class:`tiktoken.Encoding`, producing byte-identical output on all
+        hot-path methods.
+
+    Raises
+    ------
+    ImportError
+        If the ``tiktoken`` package is not installed. Install it with
+        ``pip install tiktoken``, or construct :class:`CoreBPE` directly
+        from a ``.tiktoken`` file via :func:`load_tiktoken_bpe`.
+    """
+    try:
+        import tiktoken
+    except ImportError as e:
+        raise ImportError(
+            "riptoken.get_encoding requires the `tiktoken` package to supply "
+            "vocabulary files and regex patterns. Install it with "
+            "`pip install tiktoken`, or construct CoreBPE directly using "
+            "riptoken.load_tiktoken_bpe()."
+        ) from e
+    return _build_encoding(tiktoken.get_encoding(name))
+
+
+def encoding_for_model(model_name: str) -> Encoding:
+    """Load the encoding used by a specific OpenAI model.
+
+    This is the riptoken equivalent of :func:`tiktoken.encoding_for_model`.
+    For example, ``encoding_for_model("gpt-4o")`` returns an ``o200k_base``
+    encoder wrapped in a :class:`riptoken.Encoding`. See
+    :func:`get_encoding` for dependency notes.
+
+    Parameters
+    ----------
+    model_name:
+        An OpenAI model name, e.g. ``"gpt-4"``, ``"gpt-4o"``, ``"gpt-3.5-turbo"``.
+
+    Returns
+    -------
+    Encoding
+        A drop-in replacement for the corresponding
+        :class:`tiktoken.Encoding`.
+
+    Raises
+    ------
+    ImportError
+        If the ``tiktoken`` package is not installed.
+    """
+    try:
+        import tiktoken
+    except ImportError as e:
+        raise ImportError(
+            "riptoken.encoding_for_model requires the `tiktoken` package to "
+            "supply vocabulary files and regex patterns. Install it with "
+            "`pip install tiktoken`, or construct CoreBPE directly using "
+            "riptoken.load_tiktoken_bpe()."
+        ) from e
+    return _build_encoding(tiktoken.encoding_for_model(model_name))

{riptoken-0.2.1 → riptoken-0.2.3}/src/lib.rs

@@ -885,10 +885,16 @@ impl CoreBPE {
         py.detach(|| self.encode_ordinary(text))
     }
 
-    #[pyo3(name = "encode")]
-    fn py_encode(&self, py: Python<'_>, text: &str, allowed_special: HashSet<String>) -> Vec<Rank> {
+    #[pyo3(name = "encode", signature = (text, allowed_special = None))]
+    fn py_encode(
+        &self,
+        py: Python<'_>,
+        text: &str,
+        allowed_special: Option<HashSet<String>>,
+    ) -> Vec<Rank> {
         py.detach(|| {
-            let allowed_refs: HashSet<&str> = allowed_special.iter().map(|s| s.as_str()).collect();
+            let allowed = allowed_special.unwrap_or_default();
+            let allowed_refs: HashSet<&str> = allowed.iter().map(|s| s.as_str()).collect();
             self.encode(text, &allowed_refs)
         })
     }
@@ -901,16 +907,17 @@ impl CoreBPE {
         })
     }
 
-    #[pyo3(name = "encode_batch")]
+    #[pyo3(name = "encode_batch", signature = (texts, allowed_special = None))]
     fn py_encode_batch(
         &self,
         py: Python<'_>,
         texts: Vec<String>,
-        allowed_special: HashSet<String>,
+        allowed_special: Option<HashSet<String>>,
     ) -> Vec<Vec<Rank>> {
         py.detach(|| {
             let refs: Vec<&str> = texts.iter().map(|s| s.as_str()).collect();
-            let allowed_refs: HashSet<&str> = allowed_special.iter().map(|s| s.as_str()).collect();
+            let allowed = allowed_special.unwrap_or_default();
+            let allowed_refs: HashSet<&str> = allowed.iter().map(|s| s.as_str()).collect();
             self.encode_batch(&refs, &allowed_refs)
         })
     }
@@ -931,6 +938,20 @@ impl CoreBPE {
         pyo3::types::PyBytes::new(py, &bytes)
     }
 
+    /// Decode tokens into a Python `str`, matching `tiktoken.Encoding.decode`.
+    ///
+    /// Invalid UTF-8 sequences (which can occur mid-stream when a multi-byte
+    /// character spans a token boundary) are replaced with U+FFFD, matching
+    /// tiktoken's default `errors="replace"` behavior. For strict decoding or
+    /// raw bytes, use [`decode_bytes`].
+    #[pyo3(name = "decode")]
+    fn py_decode(&self, py: Python<'_>, tokens: Vec<Rank>) -> String {
+        py.detach(|| {
+            let bytes = self.decode_bytes(&tokens);
+            String::from_utf8_lossy(&bytes).into_owned()
+        })
+    }
+
     #[pyo3(name = "decode_single_token_bytes")]
     fn py_decode_single_token_bytes<'py>(
         &self,

riptoken-0.2.1/python/riptoken/__init__.py

@@ -1,183 +0,0 @@
-"""riptoken — fast BPE tokenizer for LLMs.
-
-A drop-in compatible, faster reimplementation of OpenAI's tiktoken. riptoken
-reads the same ``.tiktoken`` vocabulary files and produces byte-identical
-output for every tested corpus.
-
-Quick start
------------
-
-The simplest way to get a tokenizer, if you have ``tiktoken`` installed:
-
->>> import riptoken
->>> enc = riptoken.get_encoding("o200k_base")
->>> enc.encode_ordinary("Hello, world!")
-[13225, 11, 2375, 0]
-
-Or construct a :class:`CoreBPE` directly from a ``.tiktoken`` file:
-
->>> import riptoken
->>> ranks = riptoken.load_tiktoken_bpe("o200k_base.tiktoken")
->>> special_tokens = {"<|endoftext|>": 199999}
->>> pat = r"\\w+|\\s+"  # simplified pattern; use the full o200k pattern in prod
->>> enc = riptoken.CoreBPE(ranks, special_tokens, pat)
->>> enc.decode_bytes(enc.encode_ordinary("Hello, world!"))
-b'Hello, world!'
-
-See https://github.com/daechoi/riptoken for full documentation, benchmarks,
-and the canonical ``o200k_base`` regex pattern.
-"""
-
-from __future__ import annotations
-
-import base64
-from pathlib import Path
-from typing import Union
-
-from riptoken._riptoken import CoreBPE
-
-__version__ = "0.2.1"
-__all__ = ["CoreBPE", "encoding_for_model", "get_encoding", "load_tiktoken_bpe"]
-
-
-def load_tiktoken_bpe(path: Union[str, Path]) -> dict[bytes, int]:
-    """Load a tiktoken ``.tiktoken`` vocabulary file.
-
-    The file format is one token per line::
-
-        <base64-encoded token bytes> <integer rank>
-
-    This is the same format OpenAI ships for ``cl100k_base``, ``o200k_base``,
-    etc. You can obtain vocabularies from the ``tiktoken`` package's cache
-    directory or download them directly from OpenAI's public CDN.
-
-    Parameters
-    ----------
-    path:
-        Filesystem path to the ``.tiktoken`` file.
-
-    Returns
-    -------
-    dict[bytes, int]
-        A dictionary mapping token bytes to their integer rank, suitable for
-        passing to :class:`CoreBPE`'s first positional argument.
-
-    Raises
-    ------
-    FileNotFoundError
-        If ``path`` does not exist.
-    ValueError
-        If a line in the file is malformed.
-    """
-    ranks: dict[bytes, int] = {}
-    with open(path, "rb") as f:
-        for lineno, line in enumerate(f, start=1):
-            if not line.strip():
-                continue
-            parts = line.split()
-            if len(parts) != 2:
-                raise ValueError(
-                    f"{path}: line {lineno}: expected '<b64> <rank>', got {line!r}"
-                )
-            token_b64, rank_str = parts
-            try:
-                token_bytes = base64.b64decode(token_b64)
-                rank = int(rank_str)
-            except (ValueError, base64.binascii.Error) as e:
-                raise ValueError(
-                    f"{path}: line {lineno}: failed to parse token/rank: {e}"
-                ) from e
-            ranks[token_bytes] = rank
-    return ranks
-
-
-def _wrap_tiktoken_encoding(enc: object) -> CoreBPE:
-    """Build a :class:`CoreBPE` from a :class:`tiktoken.Encoding` instance.
-
-    Reads the vocab, pattern string, and special tokens from stable (if
-    technically private) attributes that tiktoken has exposed since its
-    first release.
-    """
-    # These three attrs have been stable since tiktoken 0.3 and are what
-    # tiktoken itself passes into its own Rust constructor.
-    mergeable_ranks = enc._mergeable_ranks  # type: ignore[attr-defined]
-    special_tokens = enc._special_tokens  # type: ignore[attr-defined]
-    pat_str = enc._pat_str  # type: ignore[attr-defined]
-    return CoreBPE(mergeable_ranks, special_tokens, pat_str)
-
-
-def get_encoding(name: str) -> CoreBPE:
-    """Load a named tiktoken encoding (``"gpt2"``, ``"cl100k_base"``, ``"o200k_base"``, ...).
-
-    This is the riptoken equivalent of :func:`tiktoken.get_encoding`. It
-    soft-depends on the ``tiktoken`` package to supply the vocabulary file,
-    regex pattern, and special-token map, then wraps them in riptoken's
-    faster Rust core. Subsequent calls with the same ``name`` reuse
-    tiktoken's in-process cache, and the vocabulary download (if needed)
-    uses tiktoken's on-disk cache at ``~/.cache/tiktoken/`` (or wherever
-    ``TIKTOKEN_CACHE_DIR`` points).
-
-    Parameters
-    ----------
-    name:
-        A tiktoken encoding name, e.g. ``"gpt2"``, ``"r50k_base"``,
-        ``"p50k_base"``, ``"cl100k_base"``, ``"o200k_base"``.
-
-    Returns
-    -------
-    CoreBPE
-        A riptoken encoder producing byte-identical output to
-        ``tiktoken.get_encoding(name)``.
-
-    Raises
-    ------
-    ImportError
-        If the ``tiktoken`` package is not installed. Install it with
-        ``pip install tiktoken``, or construct :class:`CoreBPE` directly
-        from a ``.tiktoken`` file via :func:`load_tiktoken_bpe`.
-    """
-    try:
-        import tiktoken
-    except ImportError as e:
-        raise ImportError(
-            "riptoken.get_encoding requires the `tiktoken` package to supply "
-            "vocabulary files and regex patterns. Install it with "
-            "`pip install tiktoken`, or construct CoreBPE directly using "
-            "riptoken.load_tiktoken_bpe()."
-        ) from e
-    return _wrap_tiktoken_encoding(tiktoken.get_encoding(name))
-
-
-def encoding_for_model(model_name: str) -> CoreBPE:
-    """Load the encoding used by a specific OpenAI model.
-
-    This is the riptoken equivalent of :func:`tiktoken.encoding_for_model`.
-    For example, ``encoding_for_model("gpt-4o")`` returns an ``o200k_base``
-    encoder. See :func:`get_encoding` for dependency notes.
-
-    Parameters
-    ----------
-    model_name:
-        An OpenAI model name, e.g. ``"gpt-4"``, ``"gpt-4o"``, ``"gpt-3.5-turbo"``.
-
-    Returns
-    -------
-    CoreBPE
-        A riptoken encoder producing byte-identical output to
-        ``tiktoken.encoding_for_model(model_name)``.
-
-    Raises
-    ------
-    ImportError
-        If the ``tiktoken`` package is not installed.
-    """
-    try:
-        import tiktoken
-    except ImportError as e:
-        raise ImportError(
-            "riptoken.encoding_for_model requires the `tiktoken` package to "
-            "supply vocabulary files and regex patterns. Install it with "
-            "`pip install tiktoken`, or construct CoreBPE directly using "
-            "riptoken.load_tiktoken_bpe()."
-        ) from e
-    return _wrap_tiktoken_encoding(tiktoken.encoding_for_model(model_name))