tokenizers 0.5.0-x86_64-linux → 0.5.2-x86_64-linux

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 73c92c500a274ba952cfd529fc69f96613e19f2dc2acb0e608eba84e6477442f
4
- data.tar.gz: da18d3aa30c01ae75b8b50c1fa675fd6a32acabe3f14c5a30802b6078c9801e4
3
+ metadata.gz: d8f65354a37cb71e55cf35e34146ef535a929dc9d1fe02db4e5609e10e243dfe
4
+ data.tar.gz: 0a00b89865d9d7ecb7453a5554bd8a83a2f18755b8f4f6fcf3df81e880f83e4d
5
5
  SHA512:
6
- metadata.gz: 544f43731aba4981487a1e14a13b76dd627ec60e6c68f808d47b4b721a0153fe427402e845e7595ca729ab877008b6a452b32b95a851bfb1236b273c3e64d10b
7
- data.tar.gz: 38ca9c1e9dd511f0a875de4ba83df3015186c9e609d740525da9ee7660db70e90e1f539c20e38c43ae44fbe7df09e8ffc0781de8e7826fbe0f7a8f5c86bd6125
6
+ metadata.gz: f36d33f63a215b79319ff6570e8fab37587bc4fd5c69e8f4052cd642eaa64bb8d81bcca8d526186e7e124f124bd2d184795cf2ca2c2171159b743a456e42c527
7
+ data.tar.gz: b75812b4d6f99d7f43e246565bf7f061fc71db127db564bd459d2d06ec676ae9c4233807bde6e4f168b1d5df0cd3d1cb4c6f44d3968a762cc3aead2325ea2e29
data/CHANGELOG.md CHANGED
@@ -1,3 +1,15 @@
1
+ ## 0.5.2 (2024-08-26)
2
+
3
+ - Added `from_str` method to `Tokenizer`
4
+ - Added `model` and `model=` methods to `Tokenizer`
5
+ - Added `decoder`, `pre_tokenizer`, `post_processor`, and `normalizer` methods to `Tokenizer`
6
+ - Added `decode` method to `Decoder`
7
+
8
+ ## 0.5.1 (2024-08-13)
9
+
10
+ - Updated Tokenizers to 0.20.0
11
+ - Added precompiled gem for Linux ARM MUSL
12
+
1
13
  ## 0.5.0 (2024-05-21)
2
14
 
3
15
  - Updated Tokenizers to 0.19.1