tokenizers 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (12) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +3 -0
  3. data/Cargo.lock +1891 -0
  4. data/Cargo.toml +14 -0
  5. data/LICENSE.txt +202 -0
  6. data/README.md +67 -0
  7. data/ext/tokenizers/extconf.rb +7 -0
  8. data/lib/tokenizers/char_bpe_tokenizer.rb +19 -0
  9. data/lib/tokenizers/version.rb +3 -0
  10. data/lib/tokenizers.rb +14 -0
  11. data/src/lib.rs +290 -0
  12. metadata +53 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 0ebf55c36c1e0f55d94f5ee321a3c9f00122126912b3dfa0b52f947f172344f5
4
+ data.tar.gz: 47288dd344a36f5f3a27e206300ae11c0ec5cc1f4fbf1945a3cb3718f3e13128
5
+ SHA512:
6
+ metadata.gz: 5ab6931f3a1eef124c5f071604f860ee86341ba8a50fdb1e79d13f751a8fae9c435baae3b5ea7fd34bceba2b960d368d04740922c8515ae8d3e33b5c6bdf1c22
7
+ data.tar.gz: c39ad4aef20f1ec8890bc0a0138a80b82724f49a689224cb6e02658120b84a38aa33f771c757fe3259a2ff70d9d6a1183afcbd76f9ba5335e46ccbd455eea84f
data/CHANGELOG.md ADDED
@@ -0,0 +1,3 @@
1
+ ## 0.1.0 (2022-03-19)
2
+
3
+ - First release