tokenizers 0.1.3 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/Cargo.lock +478 -400
- data/Cargo.toml +9 -12
- data/README.md +1 -3
- data/ext/tokenizers/Cargo.toml +17 -0
- data/ext/tokenizers/extconf.rb +3 -10
- data/ext/tokenizers/src/decoders.rs +14 -0
- data/ext/tokenizers/src/encoding.rs +16 -0
- data/ext/tokenizers/src/error.rs +16 -0
- data/ext/tokenizers/src/lib.rs +64 -0
- data/ext/tokenizers/src/models.rs +19 -0
- data/ext/tokenizers/src/normalizers.rs +14 -0
- data/ext/tokenizers/src/pre_tokenizers.rs +14 -0
- data/ext/tokenizers/src/tokenizer.rs +85 -0
- data/lib/tokenizers/version.rb +1 -1
- data/lib/tokenizers.rb +6 -2
- metadata +27 -5
- data/src/lib.rs +0 -290
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9d21b58f719daca2144ea63a8a89d401990417572f4748e8fcfe2398725d9055
|
|
4
|
+
data.tar.gz: d7c823a3b89e371789cf1e549714654ee55e06834b511dcd69cb0af35b7374cd
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7a099ed56473f61e34edd1f70812a2d21a75112eaa8a60fd11ae43ccffeaf93204cd9d717bd43b51251d764153ad41586524d618638c7d1a8adfea68bd5efb1e
|
|
7
|
+
data.tar.gz: 9466dea49e08af1e77761edad22b4900f25d9b3d2aa948919ea35dbfe09d23ed86dcbe31e5163e75e57bd4a2c0a06be55578eb32217d637e625a1a845ed16ea9
|
data/CHANGELOG.md
CHANGED