gigatoken 0.1.0-aarch64-linux → 0.1.1-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +11 -9
- data/lib/gigatoken/3.3/gigatoken_rb.so +0 -0
- data/lib/gigatoken/3.4/gigatoken_rb.so +0 -0
- data/lib/gigatoken/4.0/gigatoken_rb.so +0 -0
- data/lib/gigatoken/version.rb +1 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: fec464409d36594817dc67fac65e7ccdc9baac8eceeb8d80eae62d7985fa58e1
|
|
4
|
+
data.tar.gz: 86de8bbbbd4dd0cc4eb2e2b3ce9d6dba44abc12deba382c22554d27e8893bacb
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2662b1945d1a79d1adac6c8be8bc772b358eff14df211014b2890e0f266054f916928c405bb3ddca12ba6834a3cb7bac404a504d72572dd0fae750b8b9c02b6b
|
|
7
|
+
data.tar.gz: f17356a59acb9c12aee785a4d77d8b465d615eab2dbdeb297d93e4b015d8203bccdf155d44136075776dc9df80d982cef90b79a3f183848f7ed7cefd22d0e551
|
data/README.md
CHANGED
|
@@ -1,19 +1,21 @@
|
|
|
1
1
|
# gigatoken-rb
|
|
2
2
|
|
|
3
|
-
**12 GB/s / 2.8 billion tokens per second in Ruby.**
|
|
3
|
+
**12.4 GB/s / 2.8 billion tokens per second in Ruby.**
|
|
4
4
|
|
|
5
|
-
Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken), the fastest open-source BPE tokenizer around
|
|
5
|
+
Zero-copy Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken), the fastest open-source BPE tokenizer around.
|
|
6
6
|
|
|
7
7
|
| | Corpus | MB/s (median) | Gtok/s (median) |
|
|
8
8
|
|---|---|---|---|
|
|
9
|
-
| **gigatoken** (this gem, Ruby) | 11.9 GB | **12,
|
|
10
|
-
| gigatoken (Python wheel
|
|
9
|
+
| **gigatoken-rb** (this gem, Ruby) | 11.9 GB | **12,449** | **2.82** |
|
|
10
|
+
| gigatoken (Python wheel + [#38](https://github.com/marcelroed/gigatoken/issues/38)) | 11.9 GB | 12,226 | 2.77 |
|
|
11
11
|
| tiktoken (Python) | 1.35 GB | 69.7 | 0.0158 |
|
|
12
12
|
| tiktoken_ruby | 1.35 GB | 30.7 | 0.0070 |
|
|
13
13
|
| tokenizers gem (ankane) | 1.35 GB | 10.0 | 0.0023 |
|
|
14
14
|
| tokenizers (Python, Hugging Face) | 1.35 GB | 5.6 | 0.0013 |
|
|
15
15
|
|
|
16
|
-
Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster.
|
|
16
|
+
Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster. The Python row includes a fix for [marcelroed/gigatoken#38](https://github.com/marcelroed/gigatoken/issues/38) — a hidden memcpy in `shrink_to_fit()` we found while chasing the last of the Ruby–Python gap and sent upstream (one-line fix; without it the wheel lands around 7.4 GB/s).
|
|
17
|
+
|
|
18
|
+
**340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/rb/benchmarks.md](docs/rb/benchmarks.md).
|
|
17
19
|
|
|
18
20
|
## Install
|
|
19
21
|
|
|
@@ -110,13 +112,13 @@ The Ruby layer is fiber-first throughout — no `Thread`, no `Mutex`; all parall
|
|
|
110
112
|
|
|
111
113
|
This fork exists because I need fast tokenization in Ruby. The Rust core is changed as little as possible from upstream. Most of the python shell has been removed from this fork, but you can still find it [upstream](https://github.com/marcelroed/gigatoken).
|
|
112
114
|
|
|
115
|
+
SentencePiece works but — matching upstream — is less optimized than the BPE path.
|
|
116
|
+
|
|
113
117
|
Not ported/no current plans:
|
|
114
118
|
- the HF/tiktoken Python compat shims
|
|
115
119
|
- padded-batch matrices
|
|
116
120
|
- and BPE training
|
|
117
121
|
|
|
118
|
-
SentencePiece works but — matching upstream — is less optimized than the BPE path.
|
|
119
|
-
|
|
120
122
|
## Citation
|
|
121
123
|
|
|
122
124
|
The engine is Marcel Rød's gigatoken. If it shows up in your research, cite that:
|
|
@@ -132,10 +134,10 @@ The engine is Marcel Rød's gigatoken. If it shows up in your research, cite tha
|
|
|
132
134
|
|
|
133
135
|
---
|
|
134
136
|
|
|
135
|
-
<details>
|
|
137
|
+
<details open>
|
|
136
138
|
<summary>AI Use Disclosure</summary>
|
|
137
139
|
|
|
138
140
|
The Rust engine is upstream's — see <a href="https://github.com/marcelroed/gigatoken#readme">upstream's AI-use disclosure</a> for how that was built (majority hand-crafted, AI-assisted toward the end).
|
|
139
141
|
|
|
140
|
-
The Ruby port in this fork is 100% AI generated using Fable 5 and Sonnet 5 via [space-architect](https://github.com/jetpks/space-architect)
|
|
142
|
+
The Ruby port in this fork is 100% AI generated using Fable 5 and Sonnet 5 via [space-architect](https://github.com/jetpks/space-architect) in ~24 hours.
|
|
141
143
|
</details>
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
data/lib/gigatoken/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gigatoken
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.1
|
|
5
5
|
platform: aarch64-linux
|
|
6
6
|
authors:
|
|
7
7
|
- Eric Jacobs
|
|
@@ -82,6 +82,7 @@ licenses:
|
|
|
82
82
|
- MIT
|
|
83
83
|
metadata:
|
|
84
84
|
bug_tracker_uri: https://github.com/jetpks/gigatoken-rb/issues
|
|
85
|
+
changelog_uri: https://github.com/jetpks/gigatoken-rb/blob/main/CHANGELOG.md
|
|
85
86
|
homepage_uri: https://github.com/jetpks/gigatoken-rb
|
|
86
87
|
source_code_uri: https://github.com/jetpks/gigatoken-rb
|
|
87
88
|
post_install_message:
|