tokenizers 0.2.0-arm64-darwin
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +22 -0
- data/Cargo.lock +1908 -0
- data/Cargo.toml +11 -0
- data/LICENSE-THIRD-PARTY.txt +31404 -0
- data/LICENSE.txt +202 -0
- data/README.md +65 -0
- data/lib/tokenizers/2.7/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.0/tokenizers.bundle +0 -0
- data/lib/tokenizers/3.1/tokenizers.bundle +0 -0
- data/lib/tokenizers/char_bpe_tokenizer.rb +19 -0
- data/lib/tokenizers/version.rb +3 -0
- data/lib/tokenizers.rb +18 -0
- metadata +57 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: a6e10ec0abcb804829effb45e1236d854bae8d08611bac6bf4951066e45ba056
|
|
4
|
+
data.tar.gz: 9c08202c386c301eda4b4e3ad5520022a03ca4967fd9458511ef38d5e0cc33ee
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 4254620f916f75ddedc3fad6c75efe3eb74a0fbf596449526660799898e7e251f79f833812bfbfebe31f64eb8fe0988fa69d5cc8960762674a49df51b8938d3e
|
|
7
|
+
data.tar.gz: abd39a7e74866bcc1c0551522e9f18887da5e6d6c5a8c760a653c5e3b296d08268a4424b9ab3251efc9dedb55a67a5f30bf42d790e241e7f23ea3f87a988cf2f
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
## 0.2.0 (2022-12-11)
|
|
2
|
+
|
|
3
|
+
- Added precompiled gems for Linux x86-64 and Mac
|
|
4
|
+
- Switched to `rb_sys` gem for building extension
|
|
5
|
+
- Updated Tokenizers to 0.13.2
|
|
6
|
+
- Updated Rust edition to 2021
|
|
7
|
+
|
|
8
|
+
## 0.1.3 (2022-10-06)
|
|
9
|
+
|
|
10
|
+
- Updated Tokenizers to 0.13.1
|
|
11
|
+
|
|
12
|
+
## 0.1.2 (2022-09-08)
|
|
13
|
+
|
|
14
|
+
- Fixed error with installation on Linux
|
|
15
|
+
|
|
16
|
+
## 0.1.1 (2022-06-29)
|
|
17
|
+
|
|
18
|
+
- Fixed error with installation
|
|
19
|
+
|
|
20
|
+
## 0.1.0 (2022-03-19)
|
|
21
|
+
|
|
22
|
+
- First release
|