tokenizers 0.2.2-aarch64-linux
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +32 -0
- data/Cargo.lock +909 -0
- data/Cargo.toml +5 -0
- data/LICENSE-THIRD-PARTY.txt +17956 -0
- data/LICENSE.txt +202 -0
- data/README.md +65 -0
- data/lib/tokenizers/2.7/tokenizers.so +0 -0
- data/lib/tokenizers/3.0/tokenizers.so +0 -0
- data/lib/tokenizers/3.1/tokenizers.so +0 -0
- data/lib/tokenizers/3.2/tokenizers.so +0 -0
- data/lib/tokenizers/char_bpe_tokenizer.rb +19 -0
- data/lib/tokenizers/from_pretrained.rb +119 -0
- data/lib/tokenizers/version.rb +3 -0
- data/lib/tokenizers.rb +17 -0
- metadata +59 -0