gigatoken 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (7) hide show
  1. checksums.yaml +4 -4
  2. data/Cargo.lock +1 -1
  3. data/README.md +11 -9
  4. data/ext/gigatoken/Cargo.toml +1 -1
  5. data/lib/gigatoken/version.rb +1 -1
  6. data/src/batch.rs +2 -7
  7. metadata +2 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8d646f2ee02894c38fc931c9a3a0f0b24e548e12225a3df6c8c49a4c7f9eec1c
4
- data.tar.gz: 6b8fb459b7ea9869b6f6e0624e3a86ffdfe1adce176fc7bf530106f749150c13
3
+ metadata.gz: d36a860419519d55bf6d7f2fe8c30aa9c4bc25befe98ce810313758be8bf5380
4
+ data.tar.gz: 0fda91b895be4ef02de36cb16832628203bba09de6c53955997b69611e796bce
5
5
  SHA512:
6
- metadata.gz: 0a7f6f40b1450e384364ba17ef18dd40a16c408cc9d7101e941b070eefe44349eb3368db29838337f49cc3189ebaf2ae03b21c3acf9d416c10f96fb9a64ef3fa
7
- data.tar.gz: f549192a5290172dd0bf58ce1a0e26ad2baaa23fd3a4c767f199e6678a6f206aec8318935272396f4d0bfab9a954353f27145ad532616c5ce449714d0fbf4b49
6
+ metadata.gz: 65d3987a0943a5552a8e8b8493de03df9d37ac376cfc9dfc661a6c25a38cc03cd511ed4141b98580e4bd2cff64086802ad25705f9e841fb09cae779bafce0a71
7
+ data.tar.gz: 34eace98922cb0ef2cf2a4fbecc103131bdfaa26dad459254a1d458efedc28154f3b103dd3200e106f94988d2028aa66370e680c6acee284148f707a0212288b
data/Cargo.lock CHANGED
@@ -771,7 +771,7 @@ dependencies = [
771
771
 
772
772
  [[package]]
773
773
  name = "gigatoken-rb"
774
- version = "0.1.0"
774
+ version = "0.1.1"
775
775
  dependencies = [
776
776
  "gigatoken",
777
777
  "magnus",
data/README.md CHANGED
@@ -1,19 +1,21 @@
1
1
  # gigatoken-rb
2
2
 
3
- **12 GB/s / 2.8 billion tokens per second in Ruby.**
3
+ **12.4 GB/s / 2.8 billion tokens per second in Ruby.**
4
4
 
5
- Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken), the fastest open-source BPE tokenizer around — running **1.6x faster than upstream's own Python package**, on the same Rust engine.
5
+ Zero-copy Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken), the fastest open-source BPE tokenizer around.
6
6
 
7
7
  | | Corpus | MB/s (median) | Gtok/s (median) |
8
8
  |---|---|---|---|
9
- | **gigatoken** (this gem, Ruby) | 11.9 GB | **12,278** | **2.78** |
10
- | gigatoken (Python wheel, upstream) | 11.9 GB | 7,400 | 1.68 |
9
+ | **gigatoken-rb** (this gem, Ruby) | 11.9 GB | **12,449** | **2.82** |
10
+ | gigatoken (Python wheel + [#38](https://github.com/marcelroed/gigatoken/issues/38)) | 11.9 GB | 12,226 | 2.77 |
11
11
  | tiktoken (Python) | 1.35 GB | 69.7 | 0.0158 |
12
12
  | tiktoken_ruby | 1.35 GB | 30.7 | 0.0070 |
13
13
  | tokenizers gem (ankane) | 1.35 GB | 10.0 | 0.0023 |
14
14
  | tokenizers (Python, Hugging Face) | 1.35 GB | 5.6 | 0.0013 |
15
15
 
16
- Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster. **340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology, exact counts, and the caveats that matter: [docs/rb/benchmarks.md](docs/rb/benchmarks.md).
16
+ Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster. The Python row includes a fix for [marcelroed/gigatoken#38](https://github.com/marcelroed/gigatoken/issues/38) — a hidden memcpy in `shrink_to_fit()` we found while chasing the last of the Ruby–Python gap and sent upstream (one-line fix; without it the wheel lands around 7.4 GB/s).
17
+
18
+ **340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/rb/benchmarks.md](docs/rb/benchmarks.md).
17
19
 
18
20
  ## Install
19
21
 
@@ -110,13 +112,13 @@ The Ruby layer is fiber-first throughout — no `Thread`, no `Mutex`; all parall
110
112
 
111
113
  This fork exists because I need fast tokenization in Ruby. The Rust core is changed as little as possible from upstream. Most of the python shell has been removed from this fork, but you can still find it [upstream](https://github.com/marcelroed/gigatoken).
112
114
 
115
+ SentencePiece works but — matching upstream — is less optimized than the BPE path.
116
+
113
117
  Not ported/no current plans:
114
118
  - the HF/tiktoken Python compat shims
115
119
  - padded-batch matrices
116
120
  - and BPE training
117
121
 
118
- SentencePiece works but — matching upstream — is less optimized than the BPE path.
119
-
120
122
  ## Citation
121
123
 
122
124
  The engine is Marcel Rød's gigatoken. If it shows up in your research, cite that:
@@ -132,10 +134,10 @@ The engine is Marcel Rød's gigatoken. If it shows up in your research, cite tha
132
134
 
133
135
  ---
134
136
 
135
- <details>
137
+ <details open>
136
138
  <summary>AI Use Disclosure</summary>
137
139
 
138
140
  The Rust engine is upstream's — see <a href="https://github.com/marcelroed/gigatoken#readme">upstream's AI-use disclosure</a> for how that was built (majority hand-crafted, AI-assisted toward the end).
139
141
 
140
- The Ruby port in this fork is 100% AI generated using Fable 5 and Sonnet 5 via [space-architect](https://github.com/jetpks/space-architect) over ~24 hours.
142
+ The Ruby port in this fork is 100% AI generated using Fable 5 and Sonnet 5 via [space-architect](https://github.com/jetpks/space-architect) in ~24 hours.
141
143
  </details>
data/ext/gigatoken/Cargo.toml CHANGED
@@ -1,6 +1,6 @@
1
1
  [package]
2
2
  name = "gigatoken-rb"
3
- version = "0.1.0"
3
+ version = "0.1.1"
4
4
  edition = "2021"
5
5
 
6
6
  [lib]
data/lib/gigatoken/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Gigatoken
4
- VERSION = "0.1.0"
4
+ VERSION = "0.1.1"
5
5
  end
data/src/batch.rs CHANGED
@@ -600,13 +600,8 @@ impl Committer {
600
600
  unsafe {
601
601
  flat.set_len(total);
602
602
  }
603
- // Return the unused reservation. Large allocations trim in place on
604
- // the mainstream allocators (macOS libmalloc large entries, glibc
605
- // mmap'd chunks via mremap): pointer-stable, no copy, and the
606
- // untouched tail pages were never faulted so there is nothing to
607
- // tear down. An allocator that copies instead only costs one
608
- // memcpy; correctness is unaffected.
609
- flat.shrink_to_fit();
603
+ // The unused reservation tail is unfaulted virtual address space —
604
+ // freed with the Vec on drop, no trim needed to reclaim it.
610
605
  Some(flat)
611
606
  }
612
607
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gigatoken
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Eric Jacobs
@@ -139,6 +139,7 @@ licenses:
139
139
  - MIT
140
140
  metadata:
141
141
  bug_tracker_uri: https://github.com/jetpks/gigatoken-rb/issues
142
+ changelog_uri: https://github.com/jetpks/gigatoken-rb/blob/main/CHANGELOG.md
142
143
  homepage_uri: https://github.com/jetpks/gigatoken-rb
143
144
  source_code_uri: https://github.com/jetpks/gigatoken-rb
144
145
  rdoc_options: []