tokenizers 0.2.0 → 0.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -0
- data/Cargo.lock +105 -1104
- data/Cargo.toml +0 -6
- data/README.md +1 -1
- data/ext/tokenizers/Cargo.toml +5 -3
- data/ext/tokenizers/src/lib.rs +1 -4
- data/ext/tokenizers/src/tokenizer.rs +4 -18
- data/lib/tokenizers/from_pretrained.rb +119 -0
- data/lib/tokenizers/version.rb +1 -1
- data/lib/tokenizers.rb +2 -3
- metadata +4 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 197131371ec438d82623bc0aacb8fe82ba255904e847eeb9259358f38a7063f0
|
|
4
|
+
data.tar.gz: 42ef490120e56fbb79d847ec1eb2b0a6b0ca7aa8f2ad90c09d2053d167491350
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 0a21b4811cc9e31565209eb514e55d6b22302c350371a76205aeb3b67cf94ea6dabf85074cebd48c65f9eca56e8e750b83a1df841807e53afb1275961bca50ce
|
|
7
|
+
data.tar.gz: 222bb9d759e3a2cc00ad7a4950c821fdbad1bbf6d4413f237bcf9cdc0698c2011022890b3f306be6df3d70b05abd446ad43066851ffa6c27387ddf3191f7557d
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,13 @@
|
|
|
1
|
+
## 0.2.2 (2022-01-15)
|
|
2
|
+
|
|
3
|
+
- Added precompiled gem for Linux ARM
|
|
4
|
+
- Added `from_file` method
|
|
5
|
+
- Fixed error with precompiled gem on Linux x86-64
|
|
6
|
+
|
|
7
|
+
## 0.2.1 (2022-01-12)
|
|
8
|
+
|
|
9
|
+
- Added support for Ruby 3.2
|
|
10
|
+
|
|
1
11
|
## 0.2.0 (2022-12-11)
|
|
2
12
|
|
|
3
13
|
- Added precompiled gems for Linux x86-64 and Mac
|