j_r_r_token 1.1.0-aarch64-linux → 1.2.1-aarch64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e17685f1d81b35ff10603ae68f4e7a4ce89ba210bf6fb9312e8b1bb5199efcb4
-  data.tar.gz: 0fedd30ce5c3f0d86cbfdf7eb139b891e001a6b3de350762c5e850627d241d99
+  metadata.gz: c2555f1e69dd9efe300735434265290bc4e20c1800cd34495fc34e12c9b66e0e
+  data.tar.gz: 4beac38461a0ff6d7f9b056b6e6714e1872a78093f65c754331026d261b45e72
 SHA512:
-  metadata.gz: 6c706bfb03fb9f4f0158df2ad54446c696069aff49681bfdb373d0c506a5789af73b62638040e4c345bb5df4214959c2c64a8e1928225f8099a332c60f9d7e8a
-  data.tar.gz: 78e508b15d569fb772b9a6ee5de2e499840d9f3c50bf0d531c9d0a6fded5d65c517175011ada1b4295fa08076b63b7c1d917164e21228ff2d7aa3fd156e753b0
+  metadata.gz: 5b6f1cf2c0873563602c82e006b737a37390e33918e9b5276f57364a20bf2945a9c2219d7581da7e2dbe6c82f27482bc14ccc3e1427fbf92e16d44c58eb2e7c3
+  data.tar.gz: af1e10ac2ae4e98c986d9584b949c8e2b7db4ef1f59cdcbe73266375a3ea330b5cd3a00d6c64ad142ca8889f522c7b5765c8e168ac55b3ab55bd48ac24585e74

data/CHANGELOG.md

@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+---
+## [1.2.1] - 2026-05-13
+### Breaking Change
+- Minimum Ruby version raised from 2.7.0 to 3.0.0. Ruby 2.7 has been EOL since March 2023 and is no longer supported by upstream `rake-compiler-dock` (1.11.0+) which `rb_sys` uses for cross-compiling native gems. Production environments on Ruby 3.0+ are unaffected.
+
+### New Functionality
+- Added `o200k_harmony` tokenizer with support for `gpt-oss-20b` and `gpt-oss-120b`.
+- Added support for additional model aliases now recognized by `tiktoken-rs` 0.11: `gpt-5-mini`, `gpt-5-nano`, `gpt-5.x` decimal variants (e.g. `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.2-codex`), `gpt-4.5-*`, `o1`, `o3`, `o4-mini`, `codex-mini`, `codex-mini-latest`, and `ft:gpt-5` fine-tunes.
+- Fine-tuned model resolution (`ft:` prefix) is now handled natively by the upstream `get_tokenizer` mapping, so any future ft: bases are picked up automatically.
+
+### Change
+- Bumped `tiktoken-rs` from 0.7 to 0.11.
+- Bumped `magnus` from 0.6 to 0.8 (with the `old-api` feature enabled to preserve current bindings). Still supports Ruby >= 3.0.
+- Refactored Rust model dispatch to delegate to `tiktoken_rs::tokenizer::get_tokenizer` instead of maintaining a parallel alias list. The gem now stays in sync with upstream OpenAI tiktoken automatically on each `tiktoken-rs` bump.
+- CI matrix trimmed: dropped deprecated `macos-13`; consolidated to `ubuntu-latest`, `ubuntu-24.04-arm`, `macos-latest`, `macos-15-intel`. Added Ruby 4.0 to both test and cross-gem builds.
+
+---
+## [1.2.0] - 2026-05-13
+### Note
+- Partial release. The `arm-linux` cross-compile leg failed because upstream `rake-compiler-dock` 1.11.0+ dropped Ruby 2.7, so the published artifact set is incomplete. Please use 1.2.1 instead, which drops Ruby 2.7 from the support matrix and ships a complete set of platform gems.
+
 ---
 ## [1.1.0] - 2025-08-26
 ### New Functionality

data/README.md

@@ -6,7 +6,7 @@ JRRToken provides a high-performance, native Ruby interface for counting tokens
 
 The gem ships with pre-compiled native extensions for Linux and macOS, so your end-users don't need a Rust toolchain installed.
 
-Supports Ruby >= 2.7.0
+Supports Ruby >= 3.0.0
 
 Extensive Model Support: Includes tokenizers for all modern and legacy OpenAI models, recognizing dozens of model aliases automatically.
 
@@ -15,16 +15,16 @@ Extensive Model Support: Includes tokenizers for all modern and legacy OpenAI mo
 ## Install
 
 #### Gemfile
-`gem 'j_r_r_token', '~> 1.1'`
+`gem 'j_r_r_token', '~> 1.2'`
 
 #### Command Line
-`gem install j_r_r_token -v '~> 1.1'`
+`gem install j_r_r_token -v '~> 1.2'`
 
 ## Use
 
 ### Count tokens for a specific model (the model keyword is required)
 ```Ruby
-count = JRRToken::Tokenizer.count("hello world!", model: "gpt-5")
+count = JRRToken::Tokenizer.count("hello world!", model: "gpt-5.4")
 ```
 
 ### The gem recognizes many aliases, including older models
@@ -40,18 +40,21 @@ count = JRRToken::Tokenizer.count(1234, model: "o200k_base")
 If you provide an unsupported model name, the gem will raise an `ArgumentError`.
 
 ## Supported Models
-The gem automatically maps dozens of model names and prefixes to the correct underlying tokenizer. You don't need to know the tokenizer's base name (e.g., cl100k_base); just use the model name you're working with.
+The gem automatically maps dozens of model names and prefixes to the correct underlying tokenizer. You don't need to know the tokenizer's base name (e.g., cl100k_base); just use the model name you're working with. Model resolution is delegated to [`tiktoken-rs`](https://github.com/zurawiki/tiktoken-rs), so the alias list stays in sync with upstream.
 
-- o200k_base Models (e.g., gpt-5, gpt-4.1, gpt-4o)
-- cl100k_base Models (e.g., gpt-4, gpt-3.5)
+- o200k_harmony Models (e.g., gpt-oss-20b, gpt-oss-120b)
+- o200k_base Models (e.g., gpt-5.4, gpt-5-mini, gpt-4.5, gpt-4.1, gpt-4o, o1, o3, o4-mini, codex-mini)
+- cl100k_base Models (e.g., gpt-4, gpt-3.5-turbo, text-embedding-3-small)
 - p50k_base Models (e.g., text-davinci-003)
 - r50k_base Models (e.g., gpt-2)
 - p50k_edit Models
 
+Fine-tuned models (`ft:<base>:org:name:id`) are automatically resolved to the same tokenizer as their base model.
+
 ## Developing JRRToken Locally
 
 ### Install Ruby and Rust dependencies
-Ruby >= 2.7.0
+Ruby >= 3.0.0
 
 RubyGems >= 3.0.0
 

data/j_r_r_token.gemspec

@@ -12,7 +12,7 @@ Gem::Specification.new do |spec|
   spec.description = "JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base."
   spec.homepage = "https://github.com/LoganBresnahan/j_r_r_token"
   spec.license = "MIT"
-  spec.required_ruby_version = ">= 2.7.0"
+  spec.required_ruby_version = ">= 3.0.0"
   spec.required_rubygems_version = ">= 3.0.0"
 
   spec.metadata["allowed_push_host"] = "https://rubygems.org"

data/lib/j_r_r_token/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module JRRToken
-  VERSION = "1.1.0"
+  VERSION = "1.2.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: j_r_r_token
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.1
 platform: aarch64-linux
 authors:
 - Logan Bresnahan
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2025-08-27 00:00:00.000000000 Z
+date: 2026-05-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -66,12 +66,12 @@ files:
 - README.md
 - j_r_r_token.gemspec
 - lib/j_r_r_token.rb
-- lib/j_r_r_token/2.7/j_r_r_token.so
 - lib/j_r_r_token/3.0/j_r_r_token.so
 - lib/j_r_r_token/3.1/j_r_r_token.so
 - lib/j_r_r_token/3.2/j_r_r_token.so
 - lib/j_r_r_token/3.3/j_r_r_token.so
 - lib/j_r_r_token/3.4/j_r_r_token.so
+- lib/j_r_r_token/4.0/j_r_r_token.so
 - lib/j_r_r_token/version.rb
 - sig/j_r_r_token.rbs
 homepage: https://github.com/LoganBresnahan/j_r_r_token
@@ -90,10 +90,10 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.7'
+      version: '3.0'
   - - "<"
     - !ruby/object:Gem::Version
-      version: 3.5.dev
+      version: 4.1.dev
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="