ru_token 0.1.18-x86_64-darwin → 0.1.20-x86_64-darwin
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/README.md +15 -7
- data/lib/ru_token/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5aef65f29181f3ca27491a97c1e537dfde6e2ca2bc9fafc1c63ee3793ac017ba
|
|
4
|
+
data.tar.gz: 8aba689e05256bc7a1460638e022288679dcb3a8e2ca76b623287cfbd0b886a8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ce78b492b4f0727bbadbf7720cb2479a99eb42b0c415daa70e15f92b5583be5d4df860fe347d6b1e0a6a41c53dadd3cb16c3525eb0ee492e4aa3f5941e7877e3
|
|
7
|
+
data.tar.gz: 4df67d36ced56d78685c6568355fd50fe8fdae368613a8f188646706d55180d397c69500657bd3b4df7eb40326c23dab8a88f95e0415fbd78b0d3ad3a341789b
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
---
|
|
11
|
+
## [0.1.20] - 2025-07-31
|
|
12
|
+
### Change
|
|
13
|
+
- Remove x64-mingw32 from Gemfile.lock
|
|
14
|
+
- Update README
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
## [0.1.19] - 2025-07-31
|
|
18
|
+
### No Change
|
|
19
|
+
- Bumping to retry RubyGems deploy
|
|
20
|
+
|
|
10
21
|
---
|
|
11
22
|
## [0.1.18] - 2025-07-31
|
|
12
23
|
### Change
|
data/README.md
CHANGED
|
@@ -1,16 +1,24 @@
|
|
|
1
|
-
# RuToken 🦕
|
|
1
|
+
# RuToken [🦕](https://youtu.be/0MX31NTNrho?si=NgLE4-UL3zgiUVd7)
|
|
2
|
+
|
|
2
3
|
RuToken provides a high-performance, native Ruby interface for counting tokens using the powerful [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library. It leverages the speed of Rust to offer a fast and efficient way to calculate token counts for various OpenAI models.
|
|
3
4
|
|
|
4
5
|
The gem ships with pre-compiled native extensions for Linux and macOS, so your end-users don't need a Rust toolchain installed.
|
|
5
6
|
|
|
6
7
|
Supports Ruby >= 2.7.0
|
|
7
8
|
|
|
8
|
-
|
|
9
|
-
High Performance ⚡️: Uses a native Rust implementation for blazing-fast tokenization.
|
|
9
|
+
Extensive Model Support: Includes tokenizers for all modern and legacy OpenAI models, recognizing dozens of model aliases automatically.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
[https://rubygems.org/gems/ru_token](https://rubygems.org/gems/ru_token)
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
## Install
|
|
14
|
+
|
|
15
|
+
#### Gemfile
|
|
16
|
+
`gem 'ru_token', '~> 0.1.20'`
|
|
17
|
+
|
|
18
|
+
#### Command Line
|
|
19
|
+
`gem install ru_token -v '~> 0.1.20'`
|
|
20
|
+
|
|
21
|
+
## Use
|
|
14
22
|
|
|
15
23
|
### Count tokens for a specific model (the model keyword is required)
|
|
16
24
|
```Ruby
|
|
@@ -19,7 +27,7 @@ count = RuToken::Tokenizer.count("hello world!", model: "gpt-4.1")
|
|
|
19
27
|
|
|
20
28
|
### The gem recognizes many aliases, including older models
|
|
21
29
|
```Ruby
|
|
22
|
-
count = RuToken::Tokenizer.count(
|
|
30
|
+
count = RuToken::Tokenizer.count("hello world!", model: "text-davinci-003")
|
|
23
31
|
```
|
|
24
32
|
|
|
25
33
|
### The count method calls .to_s on the argument if it is not a string
|
|
@@ -57,7 +65,7 @@ Specs:
|
|
|
57
65
|
Both:
|
|
58
66
|
`bundle exec rake`
|
|
59
67
|
|
|
60
|
-
|
|
68
|
+
Interactive console for experimentation: `bin/console`
|
|
61
69
|
|
|
62
70
|
## Contributing
|
|
63
71
|
|
data/lib/ru_token/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ru_token
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.20
|
|
5
5
|
platform: x86_64-darwin
|
|
6
6
|
authors:
|
|
7
7
|
- Logan Bresnahan
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2025-
|
|
11
|
+
date: 2025-08-01 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: bundler
|