tokenizers 0.4.4-x86_64-linux → 0.5.1-x86_64-linux
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Cargo.lock +189 -219
- data/LICENSE-THIRD-PARTY.txt +2006 -2476
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/3.3/tokenizers.so +0 -0
- data/lib/tokenizers/decoders/metaspace.rb +2 -2
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/pre_tokenizers/metaspace.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +3 -4
- data/lib/tokenizers/3.0/tokenizers.so +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e1ac97056f64913b16bd3d63a97b0dfa11daad03adabf7bc09fc1f1f59c3459d
|
|
4
|
+
data.tar.gz: 022cb74e0108763c23f26715accff37148809f564813409db81b9c85a8273fd4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f00e700813f6dffb830497943538d4d9d45fe4679b38a5132dd18564088fd644340b7c7f937a0ce9904f001d2914f3898cd63435ec3750c6b1b51198a5e0e207
|
|
7
|
+
data.tar.gz: 0c4b13af17bc407d788d5b263a024ccbd92130d1174ac1aa78412d2458923c9c6e32bcc562a8f68188867631321b6a33399fb594211e7f55f3964386d8f95700
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
## 0.5.1 (2024-08-13)
|
|
2
|
+
|
|
3
|
+
- Updated Tokenizers to 0.20.0
|
|
4
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
5
|
+
|
|
6
|
+
## 0.5.0 (2024-05-21)
|
|
7
|
+
|
|
8
|
+
- Updated Tokenizers to 0.19.1
|
|
9
|
+
- Replaced `add_prefix_space` with `prepend_scheme` and `split` options for `Metaspace` decoder and pre-tokenizer
|
|
10
|
+
- Dropped support for Ruby < 3.1
|
|
11
|
+
|
|
1
12
|
## 0.4.4 (2024-02-27)
|
|
2
13
|
|
|
3
14
|
- Updated Tokenizers to 0.15.2
|