tokenizers 0.5.0-x86_64-darwin → 0.5.2-x86_64-darwin
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/Cargo.lock +167 -177
- data/LICENSE-THIRD-PARTY.txt +1678 -1608
- data/lib/tokenizers/3.1/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.2/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.3/tokenizers.bundle +0 -0
- data/lib/tokenizers/from_pretrained.rb +2 -2
- data/lib/tokenizers/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 17d7c28ef87243dd719120f643f68f7cfc005b712974beb9f9513a29c3706b68
|
|
4
|
+
data.tar.gz: 218ab31a2944142938aa15f31eb0967563151d171accbb842b3d1dcf42cbca5f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c2e9f03bc0fb467a9187770c6825561c9d452ff853193fc176d5a4149fdb446f955e8aaf187a141ff98a4cbdc7d29085a291dfbd5bb9c46b684fa784b14cafb6
|
|
7
|
+
data.tar.gz: 20a3477fa09718a5f2dcf270bfb4a0343b33a3567097a24728456d73152ca19941d6eaa0c5194830d8c837d8774afb019f729f31788ec4f9f8be84c9d4e0dbc4
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
## 0.5.2 (2024-08-26)
|
|
2
|
+
|
|
3
|
+
- Added `from_str` method to `Tokenizer`
|
|
4
|
+
- Added `model` and `model=` methods to `Tokenizer`
|
|
5
|
+
- Added `decoder`, `pre_tokenizer`, `post_processor`, and `normalizer` methods to `Tokenizer`
|
|
6
|
+
- Added `decode` method to `Decoder`
|
|
7
|
+
|
|
8
|
+
## 0.5.1 (2024-08-13)
|
|
9
|
+
|
|
10
|
+
- Updated Tokenizers to 0.20.0
|
|
11
|
+
- Added precompiled gem for Linux ARM MUSL
|
|
12
|
+
|
|
1
13
|
## 0.5.0 (2024-05-21)
|
|
2
14
|
|
|
3
15
|
- Updated Tokenizers to 0.19.1
|