tokenizer-ruby 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '08ba527f839b738f57491cffd8490e9ff6eef4c34d0be6754ba987f4621d84b6'
+  data.tar.gz: 91bef2ba51e51178bd045397c4374669842e90b309d2a65d423cbc6267f2b7b7
+SHA512:
+  metadata.gz: dc5e34e66506f9f798526e36f923f75a06f27ac75e16c194bd25f5c2421270dcb2fa511b36a3e4dacd835f9036c76fd5959ace8de1291f15ebe624f9093c30e3
+  data.tar.gz: '083fb7459ff8cc61d5b50fcc085bc3dea7703b84d593004a8659d2db5214308a95a6ca41d685f78baab798ec07ef8e045e587e98340319c32757b1994afe565c'

data/CLAUDE.md

@@ -0,0 +1,186 @@
+# tokenizer-ruby
+
+## Project Overview
+
+Ruby bindings for HuggingFace's [tokenizers](https://github.com/huggingface/tokenizers) library. The tokenizers library is written in **Rust**, so we use **FFI** (not Rice) to bind to the C API that the Rust library exposes via `tokenizers-c`.
+
+The gem should allow Ruby developers to tokenize text using any HuggingFace tokenizer (GPT, BERT, LLaMA, Claude, etc.) with fast Rust-powered performance.
+
+## Author
+
+- Name: Johannes Dwi Cahyo
+- GitHub: johannesdwicahyo
+- Repo: git@github.com:johannesdwicahyo/tokenizer-ruby.git
+
+## Technical Approach
+
+### Binding Strategy: FFI via `ffi` gem
+
+Unlike zvec-ruby (which used Rice for C++ bindings), this gem wraps a **Rust** library. The recommended approach:
+
+1. Use the [`tokenizers` Rust crate](https://github.com/huggingface/tokenizers) which provides C bindings
+2. Build a small Rust cdylib that exposes a C API for the functions we need
+3. Use Ruby's `ffi` gem to call into the compiled `.dylib`/`.so`
+4. Ship precompiled binaries for common platforms (like zvec-ruby does)
+
+Alternative: use [rb-sys](https://github.com/oxidize-rb/rb-sys) + [magnus](https://github.com/matsadler/magnus) for direct Rust→Ruby bindings (more idiomatic but more complex). **Prefer this approach** if feasible — it's the modern Rust-Ruby binding standard.
+
+### If using magnus/rb-sys:
+```toml
+# Cargo.toml
+[dependencies]
+magnus = "0.7"
+tokenizers = "0.21"
+
+[lib]
+crate-type = ["cdylib"]
+```
+
+### If using FFI:
+Build a thin C wrapper around the Rust tokenizers crate, then use `ffi` gem.
+
+## Core API Design
+
+```ruby
+require "tokenizer_ruby"
+
+# Load from HuggingFace hub (downloads and caches)
+tokenizer = TokenizerRuby::Tokenizer.from_pretrained("gpt2")
+tokenizer = TokenizerRuby::Tokenizer.from_pretrained("bert-base-uncased")
+tokenizer = TokenizerRuby::Tokenizer.from_pretrained("meta-llama/Llama-3-8B")
+
+# Load from local file
+tokenizer = TokenizerRuby::Tokenizer.from_file("/path/to/tokenizer.json")
+
+# Basic encoding
+encoding = tokenizer.encode("Hello, world!")
+encoding.ids        # => [15496, 11, 995, 0]
+encoding.tokens     # => ["Hello", ",", " world", "!"]
+encoding.offsets    # => [[0,5], [5,6], [6,12], [12,13]]
+encoding.length     # => 4
+
+# Decode
+tokenizer.decode([15496, 11, 995, 0])  # => "Hello, world!"
+
+# Batch encoding
+encodings = tokenizer.encode_batch(["Hello", "World"])
+
+# Token counting (most common use case)
+tokenizer.count("Some text here")  # => 3
+
+# Truncation and padding
+tokenizer.enable_truncation(max_length: 512)
+tokenizer.enable_padding(length: 512, pad_token: "[PAD]")
+
+# Vocab
+tokenizer.vocab_size  # => 50257
+tokenizer.token_to_id("hello")  # => 31373
+tokenizer.id_to_token(31373)    # => "hello"
+```
+
+## Features to Implement
+
+### Phase 1 — Core (MVP)
+- [ ] `from_pretrained(model_name)` — download from HuggingFace hub
+- [ ] `from_file(path)` — load from local tokenizer.json
+- [ ] `encode(text)` → Encoding (ids, tokens, offsets, attention_mask)
+- [ ] `decode(ids)` → String
+- [ ] `encode_batch(texts)` → Array of Encodings
+- [ ] `decode_batch(ids_array)` → Array of Strings
+- [ ] `vocab_size`
+- [ ] `token_to_id(token)` / `id_to_token(id)`
+
+### Phase 2 — Convenience
+- [ ] `count(text)` — token count helper
+- [ ] `truncate(text, max_tokens:)` — truncate to token limit
+- [ ] `enable_truncation(max_length:)`
+- [ ] `enable_padding(length:, pad_token:)`
+- [ ] Thread safety (Rust tokenizer is Send+Sync)
+
+### Phase 3 — Rails Integration
+- [ ] `TokenizerRuby.default_tokenizer = "gpt2"` global config
+- [ ] ActiveModel validator: `validates :content, token_length: { maximum: 4096 }`
+- [ ] Caching of downloaded tokenizer files
+
+## Project Structure
+
+```
+tokenizer-ruby/
+├── CLAUDE.md
+├── Gemfile
+├── Rakefile
+├── LICENSE                  # MIT
+├── README.md
+├── tokenizer-ruby.gemspec
+├── lib/
+│   ├── tokenizer_ruby.rb
+│   ├── tokenizer_ruby/
+│   │   ├── version.rb
+│   │   ├── tokenizer.rb
+│   │   └── encoding.rb
+├── ext/
+│   └── tokenizer_ruby/
+│       ├── Cargo.toml
+│       ├── extconf.rb
+│       └── src/
+│           └── lib.rs
+├── test/
+│   ├── test_helper.rb
+│   ├── test_tokenizer.rb
+│   └── test_encoding.rb
+└── script/
+    └── package_native_gem.rb
+```
+
+## Dependencies
+
+### Runtime
+- `rb_sys` (if using magnus) OR `ffi` (if using FFI approach)
+
+### Build
+- Rust toolchain (cargo, rustc)
+- `rake-compiler` for building native extensions
+
+### Development
+- `minitest` for testing
+- `rake` for tasks
+
+## Precompiled Gems
+
+Follow the same pattern as zvec-ruby:
+- Build for x86_64-linux, aarch64-linux, x86_64-darwin, arm64-darwin
+- Ruby versions 3.1, 3.2, 3.3, 3.4
+- GitHub Actions workflow for automated builds on tag push
+- Users just `gem install tokenizer-ruby` with zero build dependencies
+
+## Key References
+
+- HuggingFace tokenizers repo: https://github.com/huggingface/tokenizers
+- tokenizers Rust crate docs: https://docs.rs/tokenizers
+- magnus (Rust-Ruby bindings): https://github.com/matsadler/magnus
+- rb-sys: https://github.com/oxidize-rb/rb-sys
+- Existing Ruby tokenizer gems (for API inspiration, not wrapping):
+  - tiktoken_ruby (OpenAI only)
+  - tokenizers (Python, the reference API)
+
+## Testing Strategy
+
+- Unit tests with known tokenizer outputs (use `gpt2` as reference tokenizer)
+- Test with multiple model tokenizers (GPT-2, BERT, LLaMA)
+- Test batch encoding/decoding
+- Test special tokens handling
+- Test unicode and multilingual text
+- Test thread safety with concurrent encoding
+
+## Publishing
+
+- RubyGems.org: `gem push tokenizer-ruby-*.gem`
+- gem.coop: `GEM_HOST_API_KEY=hjncPswY8PbGDfLPw4RMj928 gem push tokenizer-ruby-*.gem --host https://beta.gem.coop/@johannesdwicahyo`
+
+## Notes from zvec-ruby Experience
+
+- Precompiled native gems are essential for adoption — nobody wants to install build tools
+- Use `script/package_native_gem.rb` pattern for macOS builds
+- Use `rake-compiler-dock` for Linux cross-compilation
+- Test on multiple Ruby versions (3.1–3.4)
+- Keep the Ruby API clean and idiomatic — hide C/Rust complexity behind a nice DSL

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Johannes Dwi Cahyo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/ext/tokenizer_ruby/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "tokenizer_ruby"
+version = "0.1.0"
+edition = "2021"
+publish = false
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = "0.7"
+tokenizers = { version = "0.21", features = ["http"] }

data/ext/tokenizer_ruby/extconf.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require "mkmf"
+require "rb_sys/mkmf"
+
+create_rust_makefile("tokenizer_ruby/tokenizer_ruby")

data/ext/tokenizer_ruby/src/lib.rs

@@ -0,0 +1,125 @@
+use magnus::{
+    define_module, function, method, prelude::*, Error, RHash, Ruby,
+    RArray,
+};
+use std::cell::RefCell;
+use tokenizers::Tokenizer;
+
+#[magnus::wrap(class = "TokenizerRuby::InternalTokenizer", free_immediately)]
+struct RubyTokenizer(RefCell<Tokenizer>);
+
+fn from_pretrained(ruby: &Ruby, identifier: String) -> Result<RubyTokenizer, Error> {
+    let tokenizer = Tokenizer::from_pretrained(&identifier, None)
+        .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))?;
+    Ok(RubyTokenizer(RefCell::new(tokenizer)))
+}
+
+fn from_file(ruby: &Ruby, path: String) -> Result<RubyTokenizer, Error> {
+    let tokenizer = Tokenizer::from_file(&path)
+        .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))?;
+    Ok(RubyTokenizer(RefCell::new(tokenizer)))
+}
+
+fn encoding_to_hash(ruby: &Ruby, encoding: &tokenizers::Encoding) -> Result<RHash, Error> {
+    let hash = RHash::new();
+
+    let ids: Vec<i64> = encoding.get_ids().iter().map(|&id| id as i64).collect();
+    hash.aset(ruby.sym_new("ids"), RArray::from_vec(ids))?;
+
+    let tokens: Vec<String> = encoding.get_tokens().to_vec();
+    hash.aset(ruby.sym_new("tokens"), RArray::from_vec(tokens))?;
+
+    let offsets_array = RArray::new();
+    for &(start, end_) in encoding.get_offsets() {
+        offsets_array.push(RArray::from_vec(vec![start as i64, end_ as i64]))?;
+    }
+    hash.aset(ruby.sym_new("offsets"), offsets_array)?;
+
+    let mask: Vec<i64> = encoding.get_attention_mask().iter().map(|&m| m as i64).collect();
+    hash.aset(ruby.sym_new("attention_mask"), RArray::from_vec(mask))?;
+
+    Ok(hash)
+}
+
+impl RubyTokenizer {
+    fn encode(ruby: &Ruby, rb_self: &Self, text: String) -> Result<RHash, Error> {
+        let encoding = rb_self.0.borrow()
+            .encode(text.as_str(), false)
+            .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))?;
+        encoding_to_hash(ruby, &encoding)
+    }
+
+    fn decode(ruby: &Ruby, rb_self: &Self, ids: Vec<u32>) -> Result<String, Error> {
+        rb_self.0.borrow()
+            .decode(&ids, true)
+            .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))
+    }
+
+    fn encode_batch(ruby: &Ruby, rb_self: &Self, texts: Vec<String>) -> Result<RArray, Error> {
+        let inputs: Vec<&str> = texts.iter().map(|s| s.as_str()).collect();
+        let encodings = rb_self.0.borrow()
+            .encode_batch(inputs, false)
+            .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))?;
+        let result = RArray::new();
+        for encoding in &encodings {
+            result.push(encoding_to_hash(ruby, encoding)?)?;
+        }
+        Ok(result)
+    }
+
+    fn decode_batch(ruby: &Ruby, rb_self: &Self, ids_array: Vec<Vec<u32>>) -> Result<Vec<String>, Error> {
+        let refs: Vec<&[u32]> = ids_array.iter().map(|v| v.as_slice()).collect();
+        rb_self.0.borrow()
+            .decode_batch(&refs, true)
+            .map_err(|e| Error::new(ruby.exception_runtime_error(), format!("{}", e)))
+    }
+
+    fn vocab_size(&self) -> usize {
+        self.0.borrow().get_vocab_size(true)
+    }
+
+    fn token_to_id(&self, token: String) -> Option<u32> {
+        self.0.borrow().token_to_id(&token)
+    }
+
+    fn id_to_token(&self, id: u32) -> Option<String> {
+        self.0.borrow().id_to_token(id)
+    }
+
+    fn enable_truncation(&self, max_length: usize) {
+        let params = tokenizers::TruncationParams {
+            max_length,
+            ..Default::default()
+        };
+        let _ = self.0.borrow_mut().with_truncation(Some(params));
+    }
+
+    fn enable_padding(&self, length: usize, pad_token: String) {
+        let params = tokenizers::PaddingParams {
+            strategy: tokenizers::PaddingStrategy::Fixed(length),
+            pad_token,
+            ..Default::default()
+        };
+        self.0.borrow_mut().with_padding(Some(params));
+    }
+}
+
+#[magnus::init]
+fn init(ruby: &Ruby) -> Result<(), Error> {
+    let module = define_module("TokenizerRuby")?;
+
+    let class = module.define_class("InternalTokenizer", ruby.class_object())?;
+    class.define_singleton_method("from_pretrained", function!(from_pretrained, 1))?;
+    class.define_singleton_method("from_file", function!(from_file, 1))?;
+    class.define_method("_encode", method!(RubyTokenizer::encode, 1))?;
+    class.define_method("_decode", method!(RubyTokenizer::decode, 1))?;
+    class.define_method("_encode_batch", method!(RubyTokenizer::encode_batch, 1))?;
+    class.define_method("_decode_batch", method!(RubyTokenizer::decode_batch, 1))?;
+    class.define_method("vocab_size", method!(RubyTokenizer::vocab_size, 0))?;
+    class.define_method("token_to_id", method!(RubyTokenizer::token_to_id, 1))?;
+    class.define_method("id_to_token", method!(RubyTokenizer::id_to_token, 1))?;
+    class.define_method("_enable_truncation", method!(RubyTokenizer::enable_truncation, 1))?;
+    class.define_method("_enable_padding", method!(RubyTokenizer::enable_padding, 2))?;
+
+    Ok(())
+}

data/lib/tokenizer_ruby/encoding.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module TokenizerRuby
+  class Encoding
+    attr_reader :ids, :tokens, :offsets, :attention_mask
+
+    def initialize(ids:, tokens:, offsets:, attention_mask:)
+      @ids = ids
+      @tokens = tokens
+      @offsets = offsets
+      @attention_mask = attention_mask
+    end
+
+    def length
+      @ids.length
+    end
+  end
+end

data/lib/tokenizer_ruby/tokenizer.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+module TokenizerRuby
+  class Tokenizer
+    def self.from_pretrained(identifier)
+      new(InternalTokenizer.from_pretrained(identifier))
+    end
+
+    def self.from_file(path)
+      new(InternalTokenizer.from_file(path))
+    end
+
+    def encode(text)
+      result = @inner._encode(text)
+      Encoding.new(
+        ids: result[:ids],
+        tokens: result[:tokens],
+        offsets: result[:offsets],
+        attention_mask: result[:attention_mask]
+      )
+    end
+
+    def decode(ids)
+      @inner._decode(ids)
+    end
+
+    def encode_batch(texts)
+      results = @inner._encode_batch(texts)
+      results.map do |result|
+        Encoding.new(
+          ids: result[:ids],
+          tokens: result[:tokens],
+          offsets: result[:offsets],
+          attention_mask: result[:attention_mask]
+        )
+      end
+    end
+
+    def decode_batch(ids_array)
+      @inner._decode_batch(ids_array)
+    end
+
+    def vocab_size
+      @inner.vocab_size
+    end
+
+    def token_to_id(token)
+      @inner.token_to_id(token)
+    end
+
+    def id_to_token(id)
+      @inner.id_to_token(id)
+    end
+
+    def count(text)
+      encode(text).length
+    end
+
+    def truncate(text, max_tokens:)
+      encoding = encode(text)
+      return text if encoding.length <= max_tokens
+
+      truncated_ids = encoding.ids[0, max_tokens]
+      decode(truncated_ids)
+    end
+
+    def enable_truncation(max_length:)
+      @inner._enable_truncation(max_length)
+    end
+
+    def enable_padding(length:, pad_token: "[PAD]")
+      @inner._enable_padding(length, pad_token)
+    end
+
+    private
+
+    def initialize(inner)
+      @inner = inner
+    end
+  end
+end

data/lib/tokenizer_ruby/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module TokenizerRuby
+  VERSION = "0.1.0"
+end

data/lib/tokenizer_ruby.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require_relative "tokenizer_ruby/version"
+require_relative "tokenizer_ruby/encoding"
+require_relative "tokenizer_ruby/tokenizer"
+
+begin
+  RUBY_VERSION =~ /(\d+\.\d+)/
+  require "tokenizer_ruby/#{$1}/tokenizer_ruby"
+rescue LoadError
+  require "tokenizer_ruby/tokenizer_ruby"
+end
+
+module TokenizerRuby
+  class Error < StandardError; end
+end

metadata

@@ -0,0 +1,65 @@
+--- !ruby/object:Gem::Specification
+name: tokenizer-ruby
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Johannes Dwi Cahyo
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rb_sys
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.9'
+description: Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers
+  library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.
+executables: []
+extensions:
+- ext/tokenizer_ruby/extconf.rb
+extra_rdoc_files: []
+files:
+- CLAUDE.md
+- LICENSE
+- ext/tokenizer_ruby/Cargo.toml
+- ext/tokenizer_ruby/extconf.rb
+- ext/tokenizer_ruby/src/lib.rs
+- lib/tokenizer_ruby.rb
+- lib/tokenizer_ruby/encoding.rb
+- lib/tokenizer_ruby/tokenizer.rb
+- lib/tokenizer_ruby/version.rb
+homepage: https://github.com/johannesdwicahyo/tokenizer-ruby
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/johannesdwicahyo/tokenizer-ruby
+  source_code_uri: https://github.com/johannesdwicahyo/tokenizer-ruby
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Ruby bindings for HuggingFace tokenizers
+test_files: []