tokenizers 0.1.3 → 0.2.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Cargo.lock +649 -442
- data/Cargo.toml +8 -12
- data/README.md +1 -3
- data/ext/tokenizers/Cargo.toml +18 -0
- data/ext/tokenizers/extconf.rb +3 -10
- data/ext/tokenizers/src/decoders.rs +14 -0
- data/ext/tokenizers/src/encoding.rs +16 -0
- data/ext/tokenizers/src/error.rs +16 -0
- data/ext/tokenizers/src/lib.rs +64 -0
- data/ext/tokenizers/src/models.rs +19 -0
- data/ext/tokenizers/src/normalizers.rs +14 -0
- data/ext/tokenizers/src/pre_tokenizers.rs +14 -0
- data/ext/tokenizers/src/tokenizer.rs +85 -0
- data/lib/tokenizers/version.rb +1 -1
- data/lib/tokenizers.rb +6 -2
- metadata +27 -5
- data/src/lib.rs +0 -290
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1b6b4b14103c5b66a75ae1c3516b2bf35a9f5b46073a08d91757af08119644a3
|
|
4
|
+
data.tar.gz: 910f671ba45815fb8584143c7b5705fc3318b2d87a81557aa02505936587f7df
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3e58b8d5d84369ba9a64838cd947c35d55f9bc203f3411ba5310ab776208aba7249220dfb84fc95442136a13ba31fe9d6be55665ce33d13e8a1cfd7c3d806e2f
|
|
7
|
+
data.tar.gz: 1d82d2cd7d405c2541aa302211e54dede73d321f8d8f522db6cf3684f24fccb9f721b6c56cdf30e7512c86c2e954288025470eefe359d2bb14bd15d3a0a9b981
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
## 0.2.1 (2022-01-12)
|
|
2
|
+
|
|
3
|
+
- Added support for Ruby 3.2
|
|
4
|
+
|
|
5
|
+
## 0.2.0 (2022-12-11)
|
|
6
|
+
|
|
7
|
+
- Added precompiled gems for Linux x86-64 and Mac
|
|
8
|
+
- Switched to `rb_sys` gem for building extension
|
|
9
|
+
- Updated Tokenizers to 0.13.2
|
|
10
|
+
- Updated Rust edition to 2021
|
|
11
|
+
|
|
1
12
|
## 0.1.3 (2022-10-06)
|
|
2
13
|
|
|
3
14
|
- Updated Tokenizers to 0.13.1
|