tiktoken_ruby 0.0.1-arm64-darwin → 0.0.2-arm64-darwin
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Gemfile.lock +1 -1
- data/README.md +9 -6
- data/lib/tiktoken_ruby/version.rb +1 -1
- data/lib/tiktoken_ruby.rb +7 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5dc44b2939b680a8e680ea5abf328389d5118bd86c7c967f218a84ad6f9aaae8
|
|
4
|
+
data.tar.gz: c072e6039a5687cb80d5ed44389d2847660efb8d45e657557e6911e1786debb4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b6b1d7ebf494a0a7bfd48f47b2aaf16c32dedbebf4e76ad723083d1dbb10fc011f9ec573762d32808af497f11f0a64c5c163106e6176406758682667d7b7f0ed
|
|
7
|
+
data.tar.gz: a7005b258a111a16e243b2912aa72bcb7f703ab46b3da66e6c05cc9c79e8ed4f6321d03212dba622ccc29b440b4e7e2f36da93f8377cd30176c91cc574c66367
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
[](https://badge.fury.io/rb/tiktoken_ruby)
|
|
1
2
|
# tiktoken_ruby
|
|
2
3
|
|
|
3
4
|
[Tiktoken](https://github.com/openai/tiktoken) is BPE tokenizer from OpenAI used with their GPT models.
|
|
@@ -5,19 +6,21 @@ This is a wrapper around it aimed primarily at enabling accurate counts of GPT m
|
|
|
5
6
|
|
|
6
7
|
## Installation
|
|
7
8
|
|
|
8
|
-
TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
|
|
9
|
-
|
|
10
9
|
Install the gem and add to the application's Gemfile by executing:
|
|
11
10
|
|
|
12
|
-
$ bundle add
|
|
11
|
+
$ bundle add tiktoken_ruby
|
|
13
12
|
|
|
14
13
|
If bundler is not being used to manage dependencies, install the gem by executing:
|
|
15
14
|
|
|
16
|
-
$ gem install
|
|
15
|
+
$ gem install tiktoken_ruby
|
|
17
16
|
|
|
18
17
|
## Usage
|
|
19
18
|
|
|
20
|
-
|
|
19
|
+
```ruby
|
|
20
|
+
encoding = Tiktoken::Encoding.r50k_base
|
|
21
|
+
tokens = encoding.encode("Hello world!")
|
|
22
|
+
puts encoding.decode(tokens)
|
|
23
|
+
```
|
|
21
24
|
|
|
22
25
|
## Development
|
|
23
26
|
|
|
@@ -27,7 +30,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
|
27
30
|
|
|
28
31
|
## Contributing
|
|
29
32
|
|
|
30
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
|
33
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/iapark/tiktoken_ruby.
|
|
31
34
|
|
|
32
35
|
## License
|
|
33
36
|
|
data/lib/tiktoken_ruby.rb
CHANGED
|
@@ -2,7 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative "tiktoken_ruby/version"
|
|
4
4
|
require_relative "tiktoken_ruby/encoding.rb"
|
|
5
|
-
|
|
5
|
+
|
|
6
|
+
begin
|
|
7
|
+
RUBY_VERSION =~ /(\d+\.\d+)/
|
|
8
|
+
require_relative "tiktoken_ruby/#{$1}/tiktoken_ruby"
|
|
9
|
+
rescue LoadError
|
|
10
|
+
require_relative "tiktoken_ruby/tiktoken_ruby"
|
|
11
|
+
end
|
|
6
12
|
|
|
7
13
|
module Tiktoken
|
|
8
14
|
class Error < StandardError; end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: tiktoken_ruby
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.0.
|
|
4
|
+
version: 0.0.2
|
|
5
5
|
platform: arm64-darwin
|
|
6
6
|
authors:
|
|
7
7
|
- IAPark
|
|
@@ -54,7 +54,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
54
54
|
requirements:
|
|
55
55
|
- - ">="
|
|
56
56
|
- !ruby/object:Gem::Version
|
|
57
|
-
version: 3.
|
|
57
|
+
version: 3.1.0
|
|
58
58
|
requirements: []
|
|
59
59
|
rubygems_version: 3.4.4
|
|
60
60
|
signing_key:
|