tiktoken_ruby 0.0.1-arm-linux → 0.0.2-arm-linux

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 142edaafc4d2d0148b2c296327c9d94df988963cab783f04e535c648b5af2d80
4
- data.tar.gz: 37992560d2c95de0cc0f5f882b5151bf907980077b1b2b8615a69de65938f105
3
+ metadata.gz: 8fdbf5ffdeb5a1612d41589dd874e234a82b354332bf7e712230590725f2d2cc
4
+ data.tar.gz: fae7a47db76ed1082d04c8be46b558bbb3124d8f8d0f35aaf364fe7fbb9e97e4
5
5
  SHA512:
6
- metadata.gz: 5117a8baec5ac09678d98d66f872109a17fba144cf4488360c30d0d2dd0e3ba8c01685abc46957cd22e21cc3095fdcef6bff935e5e802a89a341e5ea928d8a03
7
- data.tar.gz: 9ed6665261361fb4be86131715034fb064478ba9b8ef10a7791fa2c0f98511ce7241da195a3b18ba91060f3ca3daba5ed7ce50e73acd79df2c7b1aaca4584753
6
+ metadata.gz: 5f1a808d461db91243fa3d0d16fa0e1c2163f0713657498d036f4763c959cabde3f0b439dc13dfb55834e808d69879cc97ea4aae2ee8a1a540d59d0d98f69f84
7
+ data.tar.gz: e20d0ce489fe375ff4e2ec9974ea76895234a1d85ef5c8d1ce1033c7759bed36483d83545a671499365657ff4239f04569ee9f90075d49abc2fbab9a5b4369a8
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- tiktoken_ruby (0.0.1)
4
+ tiktoken_ruby (0.0.2)
5
5
 
6
6
  GEM
7
7
  remote: https://rubygems.org/
data/README.md CHANGED
@@ -1,3 +1,4 @@
1
+ [![Gem Version](https://badge.fury.io/rb/tiktoken_ruby.svg)](https://badge.fury.io/rb/tiktoken_ruby)
1
2
  # tiktoken_ruby
2
3
 
3
4
  [Tiktoken](https://github.com/openai/tiktoken) is BPE tokenizer from OpenAI used with their GPT models.
@@ -5,19 +6,21 @@ This is a wrapper around it aimed primarily at enabling accurate counts of GPT m
5
6
 
6
7
  ## Installation
7
8
 
8
- TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
9
-
10
9
  Install the gem and add to the application's Gemfile by executing:
11
10
 
12
- $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
11
+ $ bundle add tiktoken_ruby
13
12
 
14
13
  If bundler is not being used to manage dependencies, install the gem by executing:
15
14
 
16
- $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
15
+ $ gem install tiktoken_ruby
17
16
 
18
17
  ## Usage
19
18
 
20
- TODO: Write usage instructions here
19
+ ```ruby
20
+ encoding = Tiktoken::Encoding.r50k_base
21
+ tokens = encoding.encode("Hello world!")
22
+ puts encoding.decode(tokens)
23
+ ```
21
24
 
22
25
  ## Development
23
26
 
@@ -27,7 +30,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
27
30
 
28
31
  ## Contributing
29
32
 
30
- Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/tiktoken_ruby.
33
+ Bug reports and pull requests are welcome on GitHub at https://github.com/iapark/tiktoken_ruby.
31
34
 
32
35
  ## License
33
36
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tiktoken
4
- VERSION = "0.0.1"
4
+ VERSION = "0.0.2"
5
5
  end
data/lib/tiktoken_ruby.rb CHANGED
@@ -2,7 +2,13 @@
2
2
 
3
3
  require_relative "tiktoken_ruby/version"
4
4
  require_relative "tiktoken_ruby/encoding.rb"
5
- require_relative "tiktoken_ruby/tiktoken_ruby"
5
+
6
+ begin
7
+ RUBY_VERSION =~ /(\d+\.\d+)/
8
+ require_relative "tiktoken_ruby/#{$1}/tiktoken_ruby"
9
+ rescue LoadError
10
+ require_relative "tiktoken_ruby/tiktoken_ruby"
11
+ end
6
12
 
7
13
  module Tiktoken
8
14
  class Error < StandardError; end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tiktoken_ruby
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: arm-linux
6
6
  authors:
7
7
  - IAPark
@@ -54,7 +54,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
54
54
  requirements:
55
55
  - - ">="
56
56
  - !ruby/object:Gem::Version
57
- version: 3.3.11
57
+ version: 3.1.0
58
58
  requirements: []
59
59
  rubygems_version: 3.4.4
60
60
  signing_key: