j_r_r_token 1.0.0-arm64-darwin

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ceb33d317de17949aebb422787424e688042bb329d7d79234a47839419863d11
+  data.tar.gz: 4e83c6357d57fa09ff04187bd942649692e5d14316cfbea346139cae952b5b27
+SHA512:
+  metadata.gz: b8f2edcc0c534704248e58cd83606756dfbc9e4a92d1ff61f115e2ba770fe251e90e4b45bb7dcb9dfb6476b9494e0ea60596ed2387a723f0be08ed3a5a722c0b
+  data.tar.gz: 38136e2d54699ba0b8f4e16adc78684cd54809b8e13062226b5914464a6e08903c5494a93c0e2dcd9f50cda12b6e943b464197bda110546d28788b52b21bcef2

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/CHANGELOG.md

@@ -0,0 +1,140 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+---
+## [1.0.0] - 2025-08-01
+### Breaking Change
+- Rename Gem from ru_token (RuToken) to j_r_r_token (JRRToken).
+- The ru_token gem will be patched with a deprecation notice.
+- Bumping the version to 1.0.0 to symbolize the gem is now in a stable state.
+
+---
+## [0.1.20] - 2025-07-31
+### Change
+- Remove x64-mingw32 from Gemfile.lock
+- Update README
+
+---
+## [0.1.19] - 2025-07-31
+### No Change
+- Bumping to retry RubyGems deploy
+
+---
+## [0.1.18] - 2025-07-31
+### Change
+- Update README
+
+---
+## [0.1.17] - 2025-07-31
+### Change
+- Update gemspec license entry
+- Remove platform x64-mingw-ucrt from Gemfile.lock
+
+---
+## [0.1.16] - 2025-07-31
+### Change
+- Update license to MIT
+
+---
+## [0.1.15] - 2025-07-31
+### Fix
+- Stable ruby version build syntax
+- Add ruby versions manual list for build
+
+---
+## [0.1.14] - 2025-07-31
+### Fix
+- Stable ruby version build syntax
+
+---
+## [0.1.13] - 2025-07-31
+### Fix
+- Stable ruby version build syntax
+
+---
+## [0.1.12] - 2025-07-31
+### Fix
+- Set stable ruby versions manually.
+- Test macos-14 and macos-15
+
+---
+## [0.1.11] - 2025-07-31
+### Fix
+- Set stable ruby versions to default in build.
+
+---
+## [0.1.10] - 2025-07-31
+### Fix
+- Add stable ruby versions to ci data step in build.
+
+---
+## [0.1.9] - 2025-07-31
+### Fix
+- Add required ruby version to build and deploy.
+
+---
+## [0.1.8] - 2025-07-30
+### Fix
+- Correct loading paths for native gem versions.
+
+---
+## [0.1.7] - 2025-07-30
+### Changed
+- Add Generic Gem Build.
+
+---
+## [0.1.6] - 2025-07-30
+### Changed
+- Change build process to rely on Oxidize.
+
+---
+## [0.1.5] - 2025-07-30
+### Changed
+- Update Deploy Process for PKG gem names.
+
+---
+## [0.1.4] - 2025-07-30
+### Changed
+- Removed test runners from release process.
+
+---
+## [0.1.3] - 2025-07-30
+### Changed
+- Update release process to properly pre-compile native gems.
+
+---
+## [0.1.2] - 2025-07-30
+
+### Changed
+- Revamped the Linux publishing workflow to use the `oxidize-rb/actions/cross-gem` action. This produces highly portable "fat" gems for Linux, resulting in several key improvements:
+  - Pre-compiled gems are now provided for both `glibc` (standard Linux) and `musl` (Alpine Linux) systems.
+  - Both Intel (`x86_64`) and ARM (`aarch64`) architectures are now supported for both `glibc` and `musl`.
+  - Linux gems are now compatible with a much wider range of distributions, including older enterprise versions.
+
+---
+## [0.1.1] - 2025-07-30
+
+### Added
+- Comprehensive RSpec test suite to validate token counts across all supported models.
+- GitHub Actions workflow for continuous integration to automatically test against multiple Ruby versions and operating systems.
+- Standard `rake release` task for simplified and automated gem publishing.
+
+### Changed
+- Updated README with accurate usage examples and a complete, collapsible list of supported model aliases.
+- Improved the default `rake` task to automatically compile the extension and run tests.
+
+### Fixed
+- Corrected expected token counts in tests to match the `tiktoken-rs` library's actual output.
+
+---
+## [0.1.0] - 2025-07-29
+
+### Added
+- Initial release of the `RuToken` gem.
+- Core functionality to count tokens for OpenAI models using a native Rust extension.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Logan Bresnahan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,78 @@
+# JRRToken [<img src="ring.png" width="32px" alt="One ring to rule them all, one ring to find them, One ring to bring them all and in the darkness bind them.">](https://youtu.be/lemgdzLDYqA?si=Z_Ls3hbJ-Xv38oJN)
+
+(Just Ruby Rust Tokens)
+
+JRRToken provides a high-performance, native Ruby interface for counting tokens using the powerful [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library. It leverages the speed of Rust to offer a fast and efficient way to calculate token counts for various OpenAI models.
+
+The gem ships with pre-compiled native extensions for Linux and macOS, so your end-users don't need a Rust toolchain installed.
+
+Supports Ruby >= 2.7.0
+
+Extensive Model Support: Includes tokenizers for all modern and legacy OpenAI models, recognizing dozens of model aliases automatically.
+
+[https://rubygems.org/gems/j_r_r_token](https://rubygems.org/gems/j_r_r_token)
+
+## Install
+
+#### Gemfile
+`gem 'j_r_r_token', '~> 1.0.0'`
+
+#### Command Line
+`gem install j_r_r_token -v '~> 1.0.0'`
+
+## Use
+
+### Count tokens for a specific model (the model keyword is required)
+```Ruby
+count = JRRToken::Tokenizer.count("hello world!", model: "gpt-4.1")
+```
+
+### The gem recognizes many aliases, including older models
+```Ruby
+count = JRRToken::Tokenizer.count("hello world!", model: "text-davinci-003")
+```
+
+### The count method calls .to_s on the argument if it is not a string
+```Ruby
+count = JRRToken::Tokenizer.count(1234, model: "o200k_base")
+```
+
+If you provide an unsupported model name, the gem will raise an `ArgumentError`.
+
+## Supported Models
+The gem automatically maps dozens of model names and prefixes to the correct underlying tokenizer. You don't need to know the tokenizer's base name (e.g., cl100k_base); just use the model name you're working with.
+
+- o200k_base Models (e.g., GPT-4o)
+- cl100k_base Models (e.g., GPT-4, GPT-3.5)
+- p50k_base Models (e.g., text-davinci-003)
+- r50k_base Models (e.g., GPT-2)
+- p50k_edit Models
+
+## Developing JRRToken Locally
+
+### Install Ruby and Rust dependencies
+Ruby >= 2.7.0
+
+RubyGems >= 3.0.0
+
+Rust edition 2021
+
+### Compile the Rust extension and run tests
+Compile:
+`bundle exec rake compile`
+
+Specs:
+`bundle exec rake spec`
+
+Both:
+`bundle exec rake`
+
+Interactive console for experimentation: `bin/console`
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/LoganBresnahan/j_r_r_token.
+
+## License
+
+The gem is available as open source under the terms of the MIT License.

data/Rakefile

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/extensiontask"
+require "rb_sys/extensiontask"
+
+GEMSPEC = Gem::Specification.load("j_r_r_token.gemspec")
+
+RbSys::ExtensionTask.new("j_r_r_token", GEMSPEC) do |ext|
+  ext.lib_dir = "lib/j_r_r_token"
+end
+
+# Only require rspec if it's available
+begin
+  require "rspec/core/rake_task"
+  RSpec::Core::RakeTask.new(:spec)
+  task default: %i[compile spec]
+rescue LoadError
+  puts "RSpec not available"
+  task default: :compile
+end
+
+task :native, [:platform] do |_t, platform:|
+  sh "bundle", "exec", "rb-sys-dock", "--platform", platform, "--build"
+end
+
+task build: :compile

data/j_r_r_token.gemspec

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require_relative "lib/j_r_r_token/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "j_r_r_token"
+  spec.version = JRRToken::VERSION
+  spec.authors = ["Logan Bresnahan"]
+  spec.email = ["loganbbres@gmail.com"]
+
+  spec.summary = "Ruby wrapper for the tiktoken Rust library, providing fast tokenization for OpenAI models."
+  spec.description = "JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base."
+  spec.homepage = "https://github.com/LoganBresnahan/j_r_r_token"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 2.7.0"
+  spec.required_rubygems_version = ">= 3.0.0"
+
+  spec.metadata["allowed_push_host"] = "https://rubygems.org"
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/main/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .circleci appveyor Gemfile])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+  spec.extensions = ["ext/j_r_r_token/Cargo.toml"]
+
+  # Development dependencies are managed here for the gem.
+  spec.add_development_dependency "bundler", ">= 2.0"
+  spec.add_development_dependency "rake", "~> 13.0"
+  spec.add_development_dependency "rake-compiler"
+
+  # For more information and examples about making a new gem, check out our
+  # guide at: https://bundler.io/guides/creating_gem.html
+end

data/lib/j_r_r_token/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module JRRToken
+  VERSION = "1.0.0"
+end

data/lib/j_r_r_token.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require_relative "j_r_r_token/version"
+
+begin
+  ruby_version = RUBY_VERSION.split('.')[0..1].join('.')
+  require "j_r_r_token/#{ruby_version}/j_r_r_token"
+rescue LoadError
+  begin
+    require "j_r_r_token/j_r_r_token"
+  rescue LoadError
+    warn "Failed to load j_r_r_token native extension."
+  end
+end
+
+module JRRToken
+  class Error < StandardError; end
+
+  # This is the main public interface for the gem.
+  class Tokenizer
+    # Counts tokens for a given text using a specified model name.
+    # The underlying Rust extension will handle mapping the model name
+    # to the correct tokenizer.
+    #
+    # @param text [String] The text to tokenize.
+    # @param model [String] The name of the model (e.g., "gpt-4o", "gpt-3.5-turbo").
+    # @return [Integer] The number of tokens.
+    def self.count(text, model:)
+      # This calls the `count_tokens` function from our Rust code,
+      # passing both the model and the text.
+      JRRToken.count_tokens(model, text.to_s)
+    end
+  end
+end

data/sig/j_r_r_token.rbs

@@ -0,0 +1,4 @@
+module JRRToken
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,111 @@
+--- !ruby/object:Gem::Specification
+name: j_r_r_token
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: arm64-darwin
+authors:
+- Logan Bresnahan
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2025-08-01 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: rake-compiler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling
+  fast and efficient tokenization for OpenAI models. It supports multiple models including
+  o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base.
+email:
+- loganbbres@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- CHANGELOG.md
+- LICENSE
+- README.md
+- Rakefile
+- j_r_r_token.gemspec
+- lib/j_r_r_token.rb
+- lib/j_r_r_token/2.7/j_r_r_token.bundle
+- lib/j_r_r_token/3.0/j_r_r_token.bundle
+- lib/j_r_r_token/3.1/j_r_r_token.bundle
+- lib/j_r_r_token/3.2/j_r_r_token.bundle
+- lib/j_r_r_token/3.3/j_r_r_token.bundle
+- lib/j_r_r_token/3.4/j_r_r_token.bundle
+- lib/j_r_r_token/version.rb
+- ring.png
+- sig/j_r_r_token.rbs
+homepage: https://github.com/LoganBresnahan/j_r_r_token
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/LoganBresnahan/j_r_r_token
+  source_code_uri: https://github.com/LoganBresnahan/j_r_r_token
+  changelog_uri: https://github.com/LoganBresnahan/j_r_r_token/blob/main/CHANGELOG.md
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '2.7'
+  - - "<"
+    - !ruby/object:Gem::Version
+      version: 3.5.dev
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+requirements: []
+rubygems_version: 3.5.23
+signing_key: 
+specification_version: 4
+summary: Ruby wrapper for the tiktoken Rust library, providing fast tokenization for
+  OpenAI models.
+test_files: []