tokenizers 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (12) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +3 -0
  3. data/Cargo.lock +1891 -0
  4. data/Cargo.toml +14 -0
  5. data/LICENSE.txt +202 -0
  6. data/README.md +67 -0
  7. data/ext/tokenizers/extconf.rb +7 -0
  8. data/lib/tokenizers/char_bpe_tokenizer.rb +19 -0
  9. data/lib/tokenizers/version.rb +3 -0
  10. data/lib/tokenizers.rb +14 -0
  11. data/src/lib.rs +290 -0
  12. metadata +53 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 0ebf55c36c1e0f55d94f5ee321a3c9f00122126912b3dfa0b52f947f172344f5
4
+ data.tar.gz: 47288dd344a36f5f3a27e206300ae11c0ec5cc1f4fbf1945a3cb3718f3e13128
5
+ SHA512:
6
+ metadata.gz: 5ab6931f3a1eef124c5f071604f860ee86341ba8a50fdb1e79d13f751a8fae9c435baae3b5ea7fd34bceba2b960d368d04740922c8515ae8d3e33b5c6bdf1c22
7
+ data.tar.gz: c39ad4aef20f1ec8890bc0a0138a80b82724f49a689224cb6e02658120b84a38aa33f771c757fe3259a2ff70d9d6a1183afcbd76f9ba5335e46ccbd455eea84f
data/CHANGELOG.md ADDED
@@ -0,0 +1,3 @@
1
+ ## 0.1.0 (2022-03-19)
2
+
3
+ - First release