tokenizers 0.4.4-aarch64-linux → 0.5.1-aarch64-linux
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Cargo.lock +189 -219
- data/LICENSE-THIRD-PARTY.txt +2006 -2476
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/3.3/tokenizers.so +0 -0
- data/lib/tokenizers/decoders/metaspace.rb +2 -2
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/pre_tokenizers/metaspace.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +3 -4
- data/lib/tokenizers/3.0/tokenizers.so +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c1dd5881d17992979c172a0b3b06d819d4d7b37f44143bf3e35c0d9e6098de56
|
|
4
|
+
data.tar.gz: e031c6b86c37c315ac94f645e73f5aef28ef1a20d649dd443cc63ebde00ac69a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d96f470e7c925bfd568db5d4ee094c9ed7ffd93d004a3f588d2596d1924deb8f38cc787459ba948909060a0395f40d93474c544920be0c974f9f4ead2d46ca0f
|
|
7
|
+
data.tar.gz: 30606972b0c329a0534304451b11a89e9f9fec4a35fb67700378aaec86186e0093c1d1b2fc3cb514eec30b3ca335442ff9c922ffe724390b335af6e24dd5901a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
## 0.5.1 (2024-08-13)
|
|
2
|
+
|
|
3
|
+
- Updated Tokenizers to 0.20.0
|
|
4
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
5
|
+
|
|
6
|
+
## 0.5.0 (2024-05-21)
|
|
7
|
+
|
|
8
|
+
- Updated Tokenizers to 0.19.1
|
|
9
|
+
- Replaced `add_prefix_space` with `prepend_scheme` and `split` options for `Metaspace` decoder and pre-tokenizer
|
|
10
|
+
- Dropped support for Ruby < 3.1
|
|
11
|
+
|
|
1
12
|
## 0.4.4 (2024-02-27)
|
|
2
13
|
|
|
3
14
|
- Updated Tokenizers to 0.15.2
|