tokenizers 0.5.0-x86_64-linux → 0.5.2-x86_64-linux
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/Cargo.lock +167 -177
- data/LICENSE-THIRD-PARTY.txt +1678 -1608
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/3.3/tokenizers.so +0 -0
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d8f65354a37cb71e55cf35e34146ef535a929dc9d1fe02db4e5609e10e243dfe
|
|
4
|
+
data.tar.gz: 0a00b89865d9d7ecb7453a5554bd8a83a2f18755b8f4f6fcf3df81e880f83e4d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f36d33f63a215b79319ff6570e8fab37587bc4fd5c69e8f4052cd642eaa64bb8d81bcca8d526186e7e124f124bd2d184795cf2ca2c2171159b743a456e42c527
|
|
7
|
+
data.tar.gz: b75812b4d6f99d7f43e246565bf7f061fc71db127db564bd459d2d06ec676ae9c4233807bde6e4f168b1d5df0cd3d1cb4c6f44d3968a762cc3aead2325ea2e29
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
## 0.5.2 (2024-08-26)
|
|
2
|
+
|
|
3
|
+
- Added `from_str` method to `Tokenizer`
|
|
4
|
+
- Added `model` and `model=` methods to `Tokenizer`
|
|
5
|
+
- Added `decoder`, `pre_tokenizer`, `post_processor`, and `normalizer` methods to `Tokenizer`
|
|
6
|
+
- Added `decode` method to `Decoder`
|
|
7
|
+
|
|
8
|
+
## 0.5.1 (2024-08-13)
|
|
9
|
+
|
|
10
|
+
- Updated Tokenizers to 0.20.0
|
|
11
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
12
|
+
|
|
1
13
|
## 0.5.0 (2024-05-21)
|
|
2
14
|
|
|
3
15
|
- Updated Tokenizers to 0.19.1
|