tokenizers 0.2.1 → 0.2.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (12) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +6 -0
  3. data/Cargo.lock +124 -1252
  4. data/Cargo.toml +0 -5
  5. data/README.md +1 -1
  6. data/ext/tokenizers/Cargo.toml +4 -3
  7. data/ext/tokenizers/src/lib.rs +1 -4
  8. data/ext/tokenizers/src/tokenizer.rs +4 -18
  9. data/lib/tokenizers/from_pretrained.rb +119 -0
  10. data/lib/tokenizers/version.rb +1 -1
  11. data/lib/tokenizers.rb +2 -3
  12. metadata +3 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1b6b4b14103c5b66a75ae1c3516b2bf35a9f5b46073a08d91757af08119644a3
4
- data.tar.gz: 910f671ba45815fb8584143c7b5705fc3318b2d87a81557aa02505936587f7df
3
+ metadata.gz: 197131371ec438d82623bc0aacb8fe82ba255904e847eeb9259358f38a7063f0
4
+ data.tar.gz: 42ef490120e56fbb79d847ec1eb2b0a6b0ca7aa8f2ad90c09d2053d167491350
5
5
  SHA512:
6
- metadata.gz: 3e58b8d5d84369ba9a64838cd947c35d55f9bc203f3411ba5310ab776208aba7249220dfb84fc95442136a13ba31fe9d6be55665ce33d13e8a1cfd7c3d806e2f
7
- data.tar.gz: 1d82d2cd7d405c2541aa302211e54dede73d321f8d8f522db6cf3684f24fccb9f721b6c56cdf30e7512c86c2e954288025470eefe359d2bb14bd15d3a0a9b981
6
+ metadata.gz: 0a21b4811cc9e31565209eb514e55d6b22302c350371a76205aeb3b67cf94ea6dabf85074cebd48c65f9eca56e8e750b83a1df841807e53afb1275961bca50ce
7
+ data.tar.gz: 222bb9d759e3a2cc00ad7a4950c821fdbad1bbf6d4413f237bcf9cdc0698c2011022890b3f306be6df3d70b05abd446ad43066851ffa6c27387ddf3191f7557d
data/CHANGELOG.md CHANGED
@@ -1,3 +1,9 @@
1
+ ## 0.2.2 (2022-01-15)
2
+
3
+ - Added precompiled gem for Linux ARM
4
+ - Added `from_file` method
5
+ - Fixed error with precompiled gem on Linux x86-64
6
+
1
7
  ## 0.2.1 (2022-01-12)
2
8
 
3
9
  - Added support for Ruby 3.2