ruby_llm-tokenizer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9c7e5f2afe7b48ff4ad5185afee11b931f499c52777ee2873e28ea232880aa72
+  data.tar.gz: 4609b223108101ac0cabd6f3fb5b9d757f95bf2e6501fe00b7d749670aa74987
+SHA512:
+  metadata.gz: 425438a6b8b8e1c2f53b79ae81bcc5b1a4eed6fcf8d9200ca0be5e17241f8d87d1e15c5cd37b2869c9fee4caa859df59e522b570c0a9f66ff241a93fa20821f2
+  data.tar.gz: a96c74f1f8619bbd8cbf1a923d19d1ea576c0cf0ec2751befa1982502a38765e1dc805128ad5957a088ebae935d96209bb6af995eee5f890b69fc20c004c6287

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/CHANGELOG.md

@@ -0,0 +1,24 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-06-05
+
+- Initial release.
+- `RubyLLM::Tokenizer.count`, `.analyze`, `.truncate` covering OpenAI (tiktoken)
+  and open-weight (Hugging Face `tokenizers`) model families.
+- `RubyLLM::Tokenizer.truncate` accepts plain strings and stream-like `Enumerable`
+  inputs (for example `File.foreach(...)`) for exact token-aware truncation.
+- `truncate` validates `max_tokens` explicitly and raises `ArgumentError` for
+  non-integer or negative values.
+- Data-driven model registry (`lib/ruby_llm/tokenizer/models.yml`) with runtime
+  `.register` for custom patterns.
+- Opt-in approximate tokenizer for Anthropic Claude via
+  `RubyLLM::Tokenizer.enable_claude_approximation!`.
+- GitHub Actions CI workflow (`.github/workflows/ci.yml`) — runs RSpec on Ruby 3.1–3.4
+  for every push and pull request; includes an enforced RuboCop lint step.
+- CI badge in README.
+- README now documents that `HUGGING_FACE_HUB_TOKEN` is accepted as an alternative to
+  `HF_TOKEN` for Hugging Face gated repository access.
+- Hugging Face tokenizers fetched from the Hub are persisted under `cache_dir` for
+  later offline reuse.
+
+

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Sal Scotto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,156 @@
+# ruby_llm-tokenizer
+
+[![CI](https://github.com/washu/ruby_llm-tokenizer/actions/workflows/ci.yml/badge.svg)](https://github.com/washu/ruby_llm-tokenizer/actions/workflows/ci.yml)
+[![Gem Version](https://badge.fury.io/rb/ruby_llm-tokenizer.svg)](https://rubygems.org/gems/ruby_llm-tokenizer)
+
+Local, model-aware token counting for [ruby_llm](https://github.com/crmne/ruby_llm).
+A pure-Ruby facade over Hugging Face [`tokenizers`](https://github.com/ankane/tokenizers-ruby) and OpenAI [`tiktoken_ruby`](https://github.com/IAPark/tiktoken_ruby) that maps model identifiers (`gpt-4o`, `llama-3`, `mistral`, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window — without making an LLM API call.
+No Rust toolchain required: cross-compiled binaries are inherited from the upstream gems.
+
+## Installation
+
+```bash
+bundle add ruby_llm-tokenizer
+```
+
+Or:
+
+```bash
+gem install ruby_llm-tokenizer
+```
+
+Requires Ruby >= 3.1.
+
+## Usage
+
+```ruby
+require "ruby_llm/tokenizer"
+
+# Count tokens
+RubyLLM::Tokenizer.count("Hello, world!", model: "gpt-4o")
+# => 4
+
+# Detailed breakdown
+analysis = RubyLLM::Tokenizer.analyze("Hello, world!", model: "gpt-4o")
+analysis.ids     # => [13225, 11, 2375, 0]
+analysis.tokens  # => ["Hello", ",", " world", "!"]
+analysis.count   # => 4
+analysis.model   # => "tiktoken:o200k_base"
+
+# Truncate to fit a context window
+RubyLLM::Tokenizer.truncate(
+  huge_log,
+  max_tokens: 30_000,
+  model: "gpt-4o",
+  overflow: :truncate_left  # drop oldest content; default is :truncate_right
+)
+
+# Stream/Enumerable inputs work too
+RubyLLM::Tokenizer.truncate(
+  File.foreach("huge_log.txt"),
+  max_tokens: 30_000,
+  model: "gpt-4o",
+  overflow: :truncate_left
+)
+```
+
+For stream-like inputs, `truncate` accepts any `Enumerable` of chunks (for example
+`File.foreach(...)`) and incrementally applies the same exact token-limit semantics as
+string input. This avoids requiring callers to materialize the original source text up
+front and avoids some duplicate tokenization work during truncation, though the
+implementation may still retain the kept portion in memory.
+
+## Supported model families (built-in)
+
+| Family                                                    | Backend         | Encoding / Repo                          |
+|-----------------------------------------------------------|-----------------|------------------------------------------|
+| All OpenAI families (gpt-3.5/4/4o/4.1/4.5/5, o-series, gpt-oss, embeddings, ft:, legacy) | `tiktoken_auto` | resolved via `Tiktoken.encoding_for_model` |
+| `llama-3` / `meta-llama`                                  | `hugging_face`  | `meta-llama/Meta-Llama-3-8B-Instruct`    |
+| `mistral` / `mixtral`                                     | `hugging_face`  | `mistralai/Mistral-7B-Instruct-v0.2`     |
+| `deepseek`                                                | `hugging_face`  | `deepseek-ai/DeepSeek-V2`                |
+| `qwen`                                                    | `hugging_face`  | `Qwen/Qwen2.5-7B-Instruct`               |
+
+OpenAI model resolution is delegated to `tiktoken_ruby` — new OpenAI models become available on `bundle update tiktoken_ruby` with no change to this gem. Override a specific model at runtime with `RubyLLM::Tokenizer.register(...)`.
+
+OpenAI encodings are bundled with `tiktoken_ruby` (no network needed). Hugging Face `tokenizer.json` files are downloaded lazily on first use, then persisted under `cache_dir` for later offline reuse. Some HF repos (Llama 3, recent Mistral) are gated and require an HF token — see [Configuration](#configuration).
+
+## Claude / Anthropic
+
+Anthropic does not publish Claude's tokenizer. By default, `model: "claude-..."` raises `UnknownModelError`.
+
+You can opt in to an approximate count (uses `o200k_base` as a stand-in; typically within 5–15% of the real number):
+
+```ruby
+RubyLLM::Tokenizer.enable_claude_approximation!
+RubyLLM::Tokenizer.count("Hello", model: "claude-3-5-sonnet-20241022")
+# warns once, then returns an approximate Integer
+```
+
+Do not use approximate counts to enforce hard context limits — leave headroom, or call Anthropic's `count_tokens` endpoint for exact numbers.
+
+## Registering custom models
+
+```ruby
+RubyLLM::Tokenizer.register(
+  match: /^my-finetuned-llama/,
+  backend: :hugging_face,
+  repo: "my-org/my-finetuned-llama-tokenizer"
+)
+
+RubyLLM::Tokenizer.register(
+  match: "gpt-4o-2024-internal",
+  backend: :tiktoken,
+  encoding: "o200k_base"
+)
+```
+
+User registrations take precedence over built-ins.
+
+## Configuration
+
+```ruby
+RubyLLM::Tokenizer.configure do |c|
+  c.cache_dir        = Pathname("/tmp/ruby_llm_tokenizer")  # default: ~/.cache/ruby_llm/tokenizer; stores downloaded HF tokenizers
+  c.offline          = false                                # if true, never hits the HF Hub
+  c.hf_token         = ENV["HF_TOKEN"]                      # also reads HUGGING_FACE_HUB_TOKEN
+  c.approximate_warn = true                                 # warn on first approximate use
+end
+```
+
+## Errors
+
+| Class                                          | Raised when                                                 |
+|------------------------------------------------|-------------------------------------------------------------|
+| `RubyLLM::Tokenizer::UnknownModelError`        | No registered pattern matches the given model id            |
+| `RubyLLM::Tokenizer::BackendError`             | Underlying tokenizer engine failed to load or encode        |
+| `RubyLLM::Tokenizer::CacheError`               | `offline: true` and the local tokenizer.json is missing     |
+| `RubyLLM::Tokenizer::ContextExceededError`     | Raised when a token count exceeds a defined limit (reserved for future use)  |
+
+
+## Development
+
+```bash
+bin/setup
+bundle exec rspec
+bin/console
+```
+
+## Releasing
+
+```bash
+SKIP_PUSH=1 ./build_release.sh
+./build_release.sh
+GEM_HOST_OTP=123456 ./build_release.sh
+```
+
+- `SKIP_PUSH=1` builds the gem and verifies the release artifact without publishing.
+- Running `./build_release.sh` normally builds and pushes, letting `gem push` prompt for MFA.
+- `GEM_HOST_OTP=...` passes an explicit RubyGems OTP when you want a non-interactive push.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/washu/ruby_llm-tokenizer.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/ruby_llm/tokenizer/analysis.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Tokenizer
+    Analysis = Struct.new(:tokens, :ids, :model, keyword_init: true) do
+      def count
+        ids.size
+      end
+
+      def to_h
+        { tokens: tokens, ids: ids, model: model, count: count }
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/backend/approximate.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require_relative "tiktoken"
+
+module RubyLLM
+  module Tokenizer
+    module Backend
+      # Approximate tokenizer for models with no published tokenizer (notably
+      # Anthropic Claude). Wraps a tiktoken encoding as a stand-in. Token counts
+      # are typically within ~5-15% of the model's true count and should not be
+      # used for hard limits.
+      class Approximate < Tiktoken
+        def initialize(encoding: "o200k_base")
+          super
+          @warned = false
+          @warn_mutex = Mutex.new
+        end
+
+        def encode(text)
+          warn_once
+          super
+        end
+
+        def identifier
+          "approximate:#{encoding_name}"
+        end
+
+        private
+
+        def warn_once
+          return if @warned
+          return unless RubyLLM::Tokenizer.configuration.approximate_warn
+
+          @warn_mutex.synchronize do
+            return if @warned
+
+            Kernel.warn(warning_message)
+            @warned = true
+          end
+        end
+
+        def warning_message
+          "[ruby_llm-tokenizer] Using approximate tokenizer (#{encoding_name}). " \
+            "Counts may differ from the model's true tokenizer by ~5-15%. " \
+            "Silence with: RubyLLM::Tokenizer.configure { |c| c.approximate_warn = false }"
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/backend/hugging_face.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "tokenizers"
+require_relative "../backend"
+
+module RubyLLM
+  module Tokenizer
+    module Backend
+      class HuggingFace < Base
+        attr_reader :repo, :revision
+
+        def initialize(repo:, revision: nil)
+          super()
+          @repo = repo
+          @revision = revision
+          @tokenizer = load_tokenizer
+        end
+
+        def encode(text)
+          @tokenizer.encode(text.to_s).ids
+        end
+
+        def decode(ids)
+          @tokenizer.decode(Array(ids), skip_special_tokens: true)
+        end
+
+        def analyze(text)
+          encoding = @tokenizer.encode(text.to_s)
+          Analysis.new(tokens: encoding.tokens, ids: encoding.ids, model: identifier)
+        end
+
+        def identifier
+          "hugging_face:#{repo}#{"@#{revision}" if revision}"
+        end
+
+        private
+
+        def load_tokenizer
+          config = RubyLLM::Tokenizer.configuration
+          config.offline ? load_from_disk(config) : load_from_hub(config)
+        rescue CacheError
+          raise
+        rescue StandardError => e
+          raise BackendError, "Failed to load Hugging Face tokenizer #{@repo.inspect}: #{e.message}"
+        end
+
+        def load_from_disk(config)
+          path = local_tokenizer_path(config)
+          raise CacheError, "Tokenizer not cached at #{path}" unless File.exist?(path)
+
+          ::Tokenizers.from_file(path.to_s)
+        end
+
+        def load_from_hub(config)
+          args = { auth_token: config.hf_token }.compact
+          args[:revision] = @revision if @revision
+          tokenizer = ::Tokenizers.from_pretrained(@repo, **args)
+          persist_to_cache(tokenizer, config)
+          tokenizer
+        end
+
+        def local_tokenizer_path(config)
+          config.cache_dir.join(@repo.tr("/", "-"), "tokenizer.json")
+        end
+
+        def persist_to_cache(tokenizer, config)
+          path = local_tokenizer_path(config)
+          FileUtils.mkdir_p(path.dirname)
+          tokenizer.save(path.to_s)
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/backend/tiktoken.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require "tiktoken_ruby"
+require_relative "../backend"
+
+module RubyLLM
+  module Tokenizer
+    module Backend
+      class Tiktoken < Base
+        attr_reader :encoding_name
+
+        def initialize(encoding:)
+          super()
+          @encoding_name = encoding.to_s
+          @encoding = ::Tiktoken.get_encoding(@encoding_name)
+          raise BackendError, "Unknown tiktoken encoding: #{encoding.inspect}" if @encoding.nil?
+        rescue StandardError => e
+          raise BackendError, "Failed to load tiktoken encoding #{encoding.inspect}: #{e.message}"
+        end
+
+        def encode(text)
+          @encoding.encode(text.to_s)
+        end
+
+        def decode(ids)
+          @encoding.decode(Array(ids))
+        end
+
+        def identifier
+          "tiktoken:#{encoding_name}"
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/backend.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require_relative "analysis"
+require_relative "errors"
+
+module RubyLLM
+  module Tokenizer
+    module Backend
+      class Base
+        def encode(_text)
+          raise NotImplementedError, "#{self.class}#encode must be implemented"
+        end
+
+        def decode(_ids)
+          raise NotImplementedError, "#{self.class}#decode must be implemented"
+        end
+
+        def count(text)
+          encode(text).size
+        end
+
+        def analyze(text)
+          ids = encode(text)
+          Analysis.new(tokens: ids.map { |id| decode_single(id) }, ids: ids, model: identifier)
+        end
+
+        def truncate(text, max_tokens:, overflow: :truncate_right)
+          validate_max_tokens!(max_tokens)
+          validate_overflow!(overflow)
+          return "" if max_tokens.zero?
+
+          if text.respond_to?(:to_str) || !text.respond_to?(:each)
+            truncate_string(text.to_s, max_tokens: max_tokens, overflow: overflow)
+          else
+            truncate_stream(text, max_tokens: max_tokens, overflow: overflow)
+          end
+        end
+
+        def identifier
+          self.class.name.split("::").last.downcase
+        end
+
+        private
+
+        def truncate_string(text, max_tokens:, overflow:)
+          ids = encode(text)
+          return text if ids.size <= max_tokens
+
+          decode(kept_ids(ids, max_tokens: max_tokens, overflow: overflow))
+        end
+
+        def truncate_stream(stream, max_tokens:, overflow:)
+          if overflow == :truncate_right
+            truncate_stream_right(stream, max_tokens)
+          else
+            truncate_stream_left(stream, max_tokens)
+          end
+        end
+
+        def truncate_stream_right(stream, max_tokens)
+          buffer = +""
+
+          each_chunk(stream) do |chunk|
+            buffer << chunk
+            ids = encode(buffer)
+            return decode(kept_ids(ids, max_tokens: max_tokens, overflow: :truncate_right)) if ids.size > max_tokens
+          end
+
+          buffer
+        end
+
+        def truncate_stream_left(stream, max_tokens)
+          buffer = +""
+
+          each_chunk(stream) do |chunk|
+            buffer << chunk
+            ids = encode(buffer)
+            buffer = decode(kept_ids(ids, max_tokens: max_tokens, overflow: :truncate_left)) if ids.size > max_tokens
+          end
+
+          buffer
+        end
+
+        def kept_ids(ids, max_tokens:, overflow:)
+          overflow == :truncate_right ? ids.first(max_tokens) : ids.last(max_tokens)
+        end
+
+        def each_chunk(stream)
+          stream.each do |chunk|
+            next if chunk.nil?
+
+            piece = chunk.to_s
+            next if piece.empty?
+
+            yield piece
+          end
+        end
+
+        def validate_max_tokens!(max_tokens)
+          raise ArgumentError, "max_tokens must be an Integer" unless max_tokens.is_a?(Integer)
+          raise ArgumentError, "max_tokens must be >= 0" if max_tokens.negative?
+        end
+
+        def validate_overflow!(overflow)
+          return if %i[truncate_right truncate_left].include?(overflow)
+
+          raise ArgumentError, "Unknown overflow strategy: #{overflow.inspect}"
+        end
+
+        def decode_single(id)
+          decode([id])
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/configuration.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "pathname"
+
+module RubyLLM
+  module Tokenizer
+    class Configuration
+      attr_accessor :cache_dir, :offline, :hf_token, :approximate_warn
+
+      def initialize
+        @cache_dir = default_cache_dir
+        @offline = false
+        @hf_token = ENV["HF_TOKEN"] || ENV.fetch("HUGGING_FACE_HUB_TOKEN", nil)
+        @approximate_warn = true
+      end
+
+      def default_cache_dir
+        base = ENV["XDG_CACHE_HOME"] || File.join(Dir.home, ".cache")
+        Pathname.new(File.join(base, "ruby_llm", "tokenizer"))
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/errors.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Tokenizer
+    class Error < StandardError; end
+
+    class UnknownModelError < Error; end
+
+    class BackendError < Error; end
+
+    class CacheError < Error; end
+
+    class ContextExceededError < Error
+      attr_reader :token_count, :limit, :model
+
+      def initialize(message = nil, token_count: nil, limit: nil, model: nil)
+        super(message)
+        @token_count = token_count
+        @limit = limit
+        @model = model
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/models.yml

@@ -0,0 +1,43 @@
+# Built-in model -> backend mapping. First match wins.
+# Users can override or extend via RubyLLM::Tokenizer.register(...).
+#
+# `match` is either a literal string (exact match) or `/regex/flags` (parsed
+# as a Regexp). Supported flags: i, m, x.
+
+# --- OpenAI: delegated to tiktoken_ruby's built-in model->encoding table ------
+# This covers every OpenAI model identifier tiktoken_ruby knows about:
+#   - GPT family: gpt-3.5, gpt-4, gpt-4o, gpt-4.1, gpt-4.5, gpt-5, gpt-oss-*,
+#                 chatgpt-4o, Azure-style gpt-35-turbo
+#   - o-series: o1, o3, o4-mini (and `-*` suffixed variants)
+#   - Embeddings: text-embedding-ada-002, text-embedding-3-{small,large}
+#   - Legacy completions: text-davinci-*, davinci/curie/babbage/ada, code-*,
+#                         davinci-002, babbage-002
+#   - Fine-tunes: ft:gpt-*, ft:davinci-002, ft:babbage-002
+#   - Codex: codex-mini, code-cushman-*, davinci-codex
+#
+# New OpenAI models become available on `bundle update tiktoken_ruby` — no PR
+# to this file required. Override a specific model at runtime with
+# RubyLLM::Tokenizer.register(...).
+
+- match: "/^(gpt-|gpt[0-9]|chatgpt-|o[1-9]|text-|code-|davinci\\b|curie\\b|babbage\\b|ada\\b|ft:|codex-)/"
+  backend: tiktoken_auto
+
+# --- Open weights: Hugging Face backend (tokenizer.json fetched lazily) -------
+# Some repos below are gated and require HF_TOKEN. Override with
+# RubyLLM::Tokenizer.register(...) if you want a different mirror.
+
+- match: "/^(llama-?3|meta-llama)/i"
+  backend: hugging_face
+  repo: meta-llama/Meta-Llama-3-8B-Instruct
+
+- match: "/^(mistral|mixtral)/i"
+  backend: hugging_face
+  repo: mistralai/Mistral-7B-Instruct-v0.2
+
+- match: "/^deepseek/i"
+  backend: hugging_face
+  repo: deepseek-ai/DeepSeek-V2
+
+- match: "/^qwen/i"
+  backend: hugging_face
+  repo: Qwen/Qwen2.5-7B-Instruct

data/lib/ruby_llm/tokenizer/registry.rb

@@ -0,0 +1,129 @@
+# frozen_string_literal: true
+
+require "yaml"
+require_relative "errors"
+require_relative "backend/tiktoken"
+require_relative "backend/hugging_face"
+require_relative "backend/approximate"
+
+module RubyLLM
+  module Tokenizer
+    class Registry
+      Entry = Struct.new(:match, :backend, :options, keyword_init: true) do
+        def matches?(model)
+          case match
+          when Regexp then match.match?(model.to_s)
+          when String then match == model.to_s
+          else false
+          end
+        end
+      end
+
+      DEFAULTS_PATH = File.join(File.dirname(__FILE__), "models.yml")
+
+      def self.load_default
+        new.tap { |r| r.load_defaults_from(DEFAULTS_PATH) }
+      end
+
+      def initialize
+        @entries = []
+        @cache = {}
+      end
+
+      # User registrations take precedence over built-ins.
+      def register(match:, backend:, **options)
+        @entries.unshift(Entry.new(match: match, backend: backend.to_sym, options: options))
+        @cache.clear
+        self
+      end
+
+      def resolve(model)
+        key = model.to_s
+        return @cache[key] if @cache.key?(key)
+
+        entry = @entries.find { |e| e.matches?(model) }
+        @cache[key] = entry && build_backend(entry, key)
+      end
+
+      def entries
+        @entries.dup
+      end
+
+      def clear
+        @entries.clear
+        @cache.clear
+        self
+      end
+
+      def load_defaults_from(path)
+        data = YAML.load_file(path) || []
+        data.each { |entry| @entries.push(build_entry(entry)) }
+        @cache.clear
+        self
+      end
+
+      def self.parse_match(value)
+        if value.is_a?(String) && (md = value.match(%r{\A/(.*)/([imx]*)\z}))
+          flags = 0
+          flags |= Regexp::IGNORECASE if md[2].include?("i")
+          flags |= Regexp::MULTILINE  if md[2].include?("m")
+          flags |= Regexp::EXTENDED   if md[2].include?("x")
+          Regexp.new(md[1], flags)
+        else
+          value
+        end
+      end
+
+      private
+
+      def build_entry(raw)
+        Entry.new(
+          match: parse_entry_match(raw),
+          backend: parse_entry_backend(raw),
+          options: entry_options(raw)
+        )
+      end
+
+      def entry_options(raw)
+        raw.except("match", "backend").transform_keys(&:to_sym)
+      end
+
+      def parse_entry_match(raw)
+        match = self.class.parse_match(raw.fetch("match"))
+        return match if match.is_a?(String) || match.is_a?(Regexp)
+
+        raise BackendError, "Invalid model matcher: #{match.inspect}"
+      end
+
+      def parse_entry_backend(raw)
+        backend = raw.fetch("backend")
+        return backend.to_sym if backend.is_a?(String) || backend.is_a?(Symbol)
+
+        raise BackendError, "Invalid backend identifier: #{backend.inspect}"
+      end
+
+      def build_backend(entry, model)
+        case entry.backend
+        when :tiktoken      then Backend::Tiktoken.new(**entry.options)
+        when :tiktoken_auto then build_tiktoken_auto(model)
+        when :hugging_face  then Backend::HuggingFace.new(**entry.options)
+        when :approximate   then Backend::Approximate.new(**entry.options)
+        else
+          raise BackendError, "Unknown backend: #{entry.backend.inspect}"
+        end
+      end
+
+      def build_tiktoken_auto(model)
+        enc = ::Tiktoken.encoding_for_model(model)
+        if enc.nil?
+          raise UnknownModelError,
+                "tiktoken_ruby has no encoding mapping for #{model.inspect}. " \
+                "Either upgrade tiktoken_ruby or register an explicit mapping with " \
+                "RubyLLM::Tokenizer.register(match: #{model.inspect}, backend: :tiktoken, encoding: \"...\")."
+        end
+
+        Backend::Tiktoken.new(encoding: enc.name)
+      end
+    end
+  end
+end

data/lib/ruby_llm/tokenizer/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Tokenizer
+    VERSION = "0.1.0"
+  end
+end

data/lib/ruby_llm/tokenizer.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require_relative "tokenizer/version"
+require_relative "tokenizer/errors"
+require_relative "tokenizer/configuration"
+require_relative "tokenizer/analysis"
+require_relative "tokenizer/registry"
+
+module RubyLLM
+  module Tokenizer
+    class << self
+      def count(text, model:)
+        backend_for(model).count(text)
+      end
+
+      def analyze(text, model:)
+        backend_for(model).analyze(text)
+      end
+
+      def truncate(text, max_tokens:, model:, overflow: :truncate_right)
+        backend_for(model).truncate(text, max_tokens: max_tokens, overflow: overflow)
+      end
+
+      def register(match:, backend:, **)
+        registry.register(match: match, backend: backend, **)
+      end
+
+      # Opt-in: route any "claude*" model to an approximation backend.
+      # Counts are not exact. See Backend::Approximate for caveats.
+      def enable_claude_approximation!(encoding: "o200k_base")
+        registry.register(match: /^claude/i, backend: :approximate, encoding: encoding)
+      end
+
+      def configure
+        yield configuration
+      end
+
+      def configuration
+        @configuration ||= Configuration.new
+      end
+
+      def registry
+        @registry ||= Registry.load_default
+      end
+
+      def reset!
+        @configuration = nil
+        @registry = nil
+      end
+
+      private
+
+      def backend_for(model)
+        registry.resolve(model) ||
+          raise(UnknownModelError, "No tokenizer configured for model: #{model.inspect}")
+      end
+    end
+  end
+end

data/sig/ruby_llm/tokenizer.rbs

@@ -0,0 +1,51 @@
+module RubyLLM
+  module Tokenizer
+    VERSION: String
+
+    class Registry
+    end
+
+    class Error < StandardError
+    end
+
+    class UnknownModelError < Error
+    end
+
+    class BackendError < Error
+    end
+
+    class CacheError < Error
+    end
+
+    class ContextExceededError < Error
+      attr_reader token_count: Integer?
+      attr_reader limit: Integer?
+      attr_reader model: String?
+    end
+
+    class Configuration
+      attr_accessor cache_dir: untyped
+      attr_accessor offline: bool
+      attr_accessor hf_token: String?
+      attr_accessor approximate_warn: bool
+    end
+
+    class Analysis
+      attr_reader tokens: Array[String]
+      attr_reader ids: Array[Integer]
+      attr_reader model: String
+      def count: () -> Integer
+      def to_h: () -> Hash[Symbol, untyped]
+    end
+
+    def self.count: (String text, model: String) -> Integer
+    def self.analyze: (String text, model: String) -> Analysis
+    def self.truncate: ((String | ::Enumerable[untyped]) text, max_tokens: Integer, model: String, ?overflow: Symbol) -> String
+    def self.register: (match: Regexp | String, backend: Symbol, **untyped) -> Registry
+    def self.enable_claude_approximation!: (?encoding: String) -> Registry
+    def self.configure: () { (Configuration) -> void } -> void
+    def self.configuration: () -> Configuration
+    def self.registry: () -> Registry
+    def self.reset!: () -> void
+  end
+end

metadata

@@ -0,0 +1,95 @@
+--- !ruby/object:Gem::Specification
+name: ruby_llm-tokenizer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Sal Scotto
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: tiktoken_ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.9
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.9
+- !ruby/object:Gem::Dependency
+  name: tokenizers
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+description: |
+  Pure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby`
+  that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the
+  correct tokenizer and exposes a small API for counting, analyzing, and
+  truncating text against a model's context window. Includes an opt-in
+  approximation backend for models with no published tokenizer (Claude).
+email:
+- sal.scotto@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/ruby_llm/tokenizer.rb
+- lib/ruby_llm/tokenizer/analysis.rb
+- lib/ruby_llm/tokenizer/backend.rb
+- lib/ruby_llm/tokenizer/backend/approximate.rb
+- lib/ruby_llm/tokenizer/backend/hugging_face.rb
+- lib/ruby_llm/tokenizer/backend/tiktoken.rb
+- lib/ruby_llm/tokenizer/configuration.rb
+- lib/ruby_llm/tokenizer/errors.rb
+- lib/ruby_llm/tokenizer/models.yml
+- lib/ruby_llm/tokenizer/registry.rb
+- lib/ruby_llm/tokenizer/version.rb
+- sig/ruby_llm/tokenizer.rbs
+homepage: https://github.com/washu/ruby_llm-tokenizer
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/washu/ruby_llm-tokenizer
+  source_code_uri: https://github.com/washu/ruby_llm-tokenizer/tree/main
+  changelog_uri: https://github.com/washu/ruby_llm-tokenizer/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.3.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Local, model-aware token counting for ruby_llm.
+test_files: []