tokenizers 0.2.1-x86_64-darwin → 0.2.3-x86_64-darwin
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +13 -0
- data/Cargo.lock +125 -1253
- data/Cargo.toml +0 -5
- data/LICENSE-THIRD-PARTY.txt +6594 -23520
- data/README.md +1 -1
- data/lib/tokenizers/2.7/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.0/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.1/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.2/tokenizers.bundle +0 -0
- data/lib/tokenizers/char_bpe_tokenizer.rb +2 -2
- data/lib/tokenizers/encoding.rb +19 -0
- data/lib/tokenizers/from_pretrained.rb +119 -0
- data/lib/tokenizers/tokenizer.rb +12 -0
- data/lib/tokenizers/version.rb +1 -1
- data/lib/tokenizers.rb +8 -7
- metadata +6 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4ccf6bfafaf681a2e5f9088b32fc41df079b48a69f865dcf7f5e3e5c9990eedb
|
|
4
|
+
data.tar.gz: 7909f67176ea9015c73e7fa76407de53c64efb6abf62972fad12504a526bb1c2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7e7bfb6230c87af3af44ed59aa45cef5fc7019b09233ba297f08e660cc2791e2e0fbbb34bd4bc5c7761198895ed56baf0df8a63c67d25379ff4821568b6ffb23
|
|
7
|
+
data.tar.gz: 834a44cc9fc48405599982477c71f3bffa52580331884b0a29b87a5d4db2fb973be0f6fd0eb7f57f70890defcd8a6486cded8c7dc6473b7b954fbd15ec0069ed
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,16 @@
|
|
|
1
|
+
## 0.2.3 (2022-01-22)
|
|
2
|
+
|
|
3
|
+
- Added `add_special_tokens` option to `encode` method
|
|
4
|
+
- Added warning about `encode` method including special tokens by default in 0.3.0
|
|
5
|
+
- Added more methods to `Encoding`
|
|
6
|
+
- Fixed error with precompiled gem on Mac ARM
|
|
7
|
+
|
|
8
|
+
## 0.2.2 (2022-01-15)
|
|
9
|
+
|
|
10
|
+
- Added precompiled gem for Linux ARM
|
|
11
|
+
- Added `from_file` method
|
|
12
|
+
- Fixed error with precompiled gem on Linux x86-64
|
|
13
|
+
|
|
1
14
|
## 0.2.1 (2022-01-12)
|
|
2
15
|
|
|
3
16
|
- Added support for Ruby 3.2
|