tokenizers 0.2.1-x86_64-linux → 0.2.3-x86_64-linux

Sign up to get free protection for your applications and to get access to all the features.
Files changed (17) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +13 -0
  3. data/Cargo.lock +125 -1253
  4. data/Cargo.toml +0 -5
  5. data/LICENSE-THIRD-PARTY.txt +6546 -23524
  6. data/README.md +1 -1
  7. data/lib/tokenizers/2.7/tokenizers.so +0 -0
  8. data/lib/tokenizers/3.0/tokenizers.so +0 -0
  9. data/lib/tokenizers/3.1/tokenizers.so +0 -0
  10. data/lib/tokenizers/3.2/tokenizers.so +0 -0
  11. data/lib/tokenizers/char_bpe_tokenizer.rb +2 -2
  12. data/lib/tokenizers/encoding.rb +19 -0
  13. data/lib/tokenizers/from_pretrained.rb +119 -0
  14. data/lib/tokenizers/tokenizer.rb +12 -0
  15. data/lib/tokenizers/version.rb +1 -1
  16. data/lib/tokenizers.rb +8 -7
  17. metadata +6 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1b91cefb36bca7ffa87d09f1ddd07ec635049d8f967ea717beb78a8175d4f4ba
4
- data.tar.gz: 3453a1b718d5b27258c99172c5cf4ed73b53ee1ec8613eb7ee1e0c3d07d22ce6
3
+ metadata.gz: 51273c1f38d9a2fcbcda6df42b1f8eff718965e84ab49233b6e54bef0825aed4
4
+ data.tar.gz: 3ce5e8543e7ac32c6302fcdefde06fdebb249be42db234bbbbe8671bb414a69a
5
5
  SHA512:
6
- metadata.gz: 971bcbc0e93172334faaeba912e4c71f088f809cc48eed55999a90511a8ad3f5e44979094dd1fc784f6a417fe46654d1465953478c446ba19dec61cd4c2903f2
7
- data.tar.gz: 2ff8a03587b8e604159d90bc0ddd6f27b6ae899ff5ea02941cec606156fef3bc62addf2876d409c40c1074e4cbe56697f0b6e6a4a5f35ff6d198896c28566d6a
6
+ metadata.gz: 45946d725ed104ca0001cf323c4bc0146050f051e8f9150aba22f5169640cef2227729ccbd9f705c43ada82ff53703ddaceb52a77259f0096066739c93c13a07
7
+ data.tar.gz: b056d1024ab43363c3ba3b0ae1cec2874fab31630de474eb0a6694484b88418bad408155121bf2adc0ed0ed4a70064765565e630d3ba648cb23e6b5c34d9aefc
data/CHANGELOG.md CHANGED
@@ -1,3 +1,16 @@
1
+ ## 0.2.3 (2022-01-22)
2
+
3
+ - Added `add_special_tokens` option to `encode` method
4
+ - Added warning about `encode` method including special tokens by default in 0.3.0
5
+ - Added more methods to `Encoding`
6
+ - Fixed error with precompiled gem on Mac ARM
7
+
8
+ ## 0.2.2 (2022-01-15)
9
+
10
+ - Added precompiled gem for Linux ARM
11
+ - Added `from_file` method
12
+ - Fixed error with precompiled gem on Linux x86-64
13
+
1
14
  ## 0.2.1 (2022-01-12)
2
15
 
3
16
  - Added support for Ruby 3.2