tokenizers 0.5.0-arm64-darwin → 0.5.2-arm64-darwin
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/Cargo.lock +167 -177
- data/LICENSE-THIRD-PARTY.txt +1678 -1608
- data/lib/tokenizers/3.1/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.2/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.3/tokenizers.bundle +0 -0
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a00c67519f2c2bb09c0fad42f396006a341be62b41d751096ef772158ed30c7d
|
|
4
|
+
data.tar.gz: 977b690303058422c76c41ebde6a8fd9981063cb88423aff535faf8a5c31d5a1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7c7ac23930faa7c58cb3c1f282c3fafb1937bc097facaaa45cfdd74f7dc0c59dee57896bfbbd5fd052e6ae3bc849affb4109dbc59a2a75ee4c4f9e644c441c65
|
|
7
|
+
data.tar.gz: 9bd2569f96710bc58aeeabd39d991ff56f13682dd86a271cc03dad2205e2a6732bb60c9eab06138405c2cf7d77df0196321d6cf7df6eac6b4f74caabb6d0a9e6
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
## 0.5.2 (2024-08-26)
|
|
2
|
+
|
|
3
|
+
- Added `from_str` method to `Tokenizer`
|
|
4
|
+
- Added `model` and `model=` methods to `Tokenizer`
|
|
5
|
+
- Added `decoder`, `pre_tokenizer`, `post_processor`, and `normalizer` methods to `Tokenizer`
|
|
6
|
+
- Added `decode` method to `Decoder`
|
|
7
|
+
|
|
8
|
+
## 0.5.1 (2024-08-13)
|
|
9
|
+
|
|
10
|
+
- Updated Tokenizers to 0.20.0
|
|
11
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
12
|
+
|
|
1
13
|
## 0.5.0 (2024-05-21)
|
|
2
14
|
|
|
3
15
|
- Updated Tokenizers to 0.19.1
|