tokenizers 0.4.4-x86_64-linux-musl → 0.5.1-x86_64-linux-musl
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Cargo.lock +189 -219
- data/LICENSE-THIRD-PARTY.txt +2006 -2476
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/3.3/tokenizers.so +0 -0
- data/lib/tokenizers/decoders/metaspace.rb +2 -2
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/pre_tokenizers/metaspace.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +3 -4
- data/lib/tokenizers/3.0/tokenizers.so +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 33811f5d5774e9715d7730def78a304df68801d3dcb044831943db2708c33c5a
|
|
4
|
+
data.tar.gz: cec6f7c1bdf1ea43014284e1716cd5d51fb4474270782706ebdb43938748df07
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a68caf15b7de717546fbedd09921d5d80357dad6354a785fa2a729cdcc78f1ea3001f001947688f4669b0e85a7db07ee147714e6c5ce2d3af103f17ddf256335
|
|
7
|
+
data.tar.gz: 8f37e091a9e7d816b949d53da9c3b7711d9e1122f380daed0f0a2fa33551e0d46d78cb7e5437e0039e5e0fc3e7834440c8460e9e600a3b1bd71a69e45490afb2
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
## 0.5.1 (2024-08-13)
|
|
2
|
+
|
|
3
|
+
- Updated Tokenizers to 0.20.0
|
|
4
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
5
|
+
|
|
6
|
+
## 0.5.0 (2024-05-21)
|
|
7
|
+
|
|
8
|
+
- Updated Tokenizers to 0.19.1
|
|
9
|
+
- Replaced `add_prefix_space` with `prepend_scheme` and `split` options for `Metaspace` decoder and pre-tokenizer
|
|
10
|
+
- Dropped support for Ruby < 3.1
|
|
11
|
+
|
|
1
12
|
## 0.4.4 (2024-02-27)
|
|
2
13
|
|
|
3
14
|
- Updated Tokenizers to 0.15.2
|