tokenizers 0.2.2-aarch64-linux
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +32 -0
- data/Cargo.lock +909 -0
- data/Cargo.toml +5 -0
- data/LICENSE-THIRD-PARTY.txt +17956 -0
- data/LICENSE.txt +202 -0
- data/README.md +65 -0
- data/lib/tokenizers/2.7/tokenizers.so +0 -0
- data/lib/tokenizers/3.0/tokenizers.so +0 -0
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/char_bpe_tokenizer.rb +19 -0
- data/lib/tokenizers/from_pretrained.rb +119 -0
- data/lib/tokenizers/version.rb +3 -0
- data/lib/tokenizers.rb +17 -0
- metadata +59 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 6eb598afb972974b72da4aabbfb72ecac6470f62e620ad4ee0cefb4aaa403173
|
|
4
|
+
data.tar.gz: a469da134839a57c31326c13721cb0f1fb34408b8b3a18a6cd16dde1b331c1a4
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 5f78428fae46491dc2414a047ee6381cb787cffe17c59ba317e48453c62e2c809abfeae7e126271f778e893aaa7a4b9ed3b90ab9fe46bec17b0392befef5d0fe
|
|
7
|
+
data.tar.gz: 6613e888311b6630a0c6ced1ba353cd5eb2c7b33ddc00c7476896c78cfe71d44d5634adb401457995d06d86d476553a343db0ae597f18c611e180c8406b9f0de
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
## 0.2.2 (2022-01-15)
|
|
2
|
+
|
|
3
|
+
- Added precompiled gem for Linux ARM
|
|
4
|
+
- Added `from_file` method
|
|
5
|
+
- Fixed error with precompiled gem on Linux x86-64
|
|
6
|
+
|
|
7
|
+
## 0.2.1 (2022-01-12)
|
|
8
|
+
|
|
9
|
+
- Added support for Ruby 3.2
|
|
10
|
+
|
|
11
|
+
## 0.2.0 (2022-12-11)
|
|
12
|
+
|
|
13
|
+
- Added precompiled gems for Linux x86-64 and Mac
|
|
14
|
+
- Switched to `rb_sys` gem for building extension
|
|
15
|
+
- Updated Tokenizers to 0.13.2
|
|
16
|
+
- Updated Rust edition to 2021
|
|
17
|
+
|
|
18
|
+
## 0.1.3 (2022-10-06)
|
|
19
|
+
|
|
20
|
+
- Updated Tokenizers to 0.13.1
|
|
21
|
+
|
|
22
|
+
## 0.1.2 (2022-09-08)
|
|
23
|
+
|
|
24
|
+
- Fixed error with installation on Linux
|
|
25
|
+
|
|
26
|
+
## 0.1.1 (2022-06-29)
|
|
27
|
+
|
|
28
|
+
- Fixed error with installation
|
|
29
|
+
|
|
30
|
+
## 0.1.0 (2022-03-19)
|
|
31
|
+
|
|
32
|
+
- First release
|