gigatoken 0.2.2 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ae6f31ee2b6996844f0f7460616ed7a3c4802993256fc68b231c7aeabf870df6
-  data.tar.gz: d5e9eb5dc28c10f26b4dbb29f04f5f2156ae1a50697687251fb2a1e1ad5f8d1b
+  metadata.gz: 41dcafd85415b4ee0ee40e7c77414557b1c6913676d4846e0d5934f689f3657a
+  data.tar.gz: fddc7d15a5a7e1b9cf9be794a0468294dc31c251ad90dd540d16e9ede344b3a2
 SHA512:
-  metadata.gz: 49ad46adee00b833739bd523905fc3f44d59374267aeff242be355246cbc173e6efdfefe8f509c287d429929df36413ccbdac487ec8a9dc6b9c5f3d758d06779
-  data.tar.gz: 80d49b5c3246c91028cb669eb4ff534f8a49af393fe974e6537b6e3fdd33c916e6c0070eaabbae8014df6d9ece96318dae1c07352568d5108426714d93d57f54
+  metadata.gz: 20b5c1fdb98dda4eff0fa0323adbf72d334ddeff672b9b1a3ae1085fdc1454ff1b00158b2aa7ea876be39188c7873c57b7df9d5aeeee99d641f3da22f88699ce
+  data.tar.gz: 1c57fc8322d4bb6531cec42def1e6194506742d9491df4b8c2daaf1fb5a1beb29877520935654a955c684d28fcab3e76ba81b5288849092dec5d217d48d190c7

data/Cargo.lock

@@ -771,7 +771,7 @@ dependencies = [
 
 [[package]]
 name = "gigatoken-rb"
-version = "0.2.2"
+version = "0.3.0"
 dependencies = [
  "gigatoken",
  "magnus",

data/README.md

@@ -15,7 +15,7 @@ Zero-copy Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed
 
 Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster. The Python row includes a fix for [marcelroed/gigatoken#38](https://github.com/marcelroed/gigatoken/issues/38) — a hidden memcpy in `shrink_to_fit()` we found while chasing the last of the Ruby–Python gap and sent upstream (one-line fix; without it the wheel lands around 7.4 GB/s).
 
-**340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/rb/benchmarks.md](docs/rb/benchmarks.md).
+**340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/explanation/benchmarks.md](docs/explanation/benchmarks.md).
 
 ## Install
 
@@ -119,7 +119,7 @@ packed[3]           # => document 3's ids as an Array, on demand
 
 ### Async
 
-`encode_batch` and `encode_files` release the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`) and the calling fiber yields to the reactor too. Design notes: [docs/rb/async.md](docs/rb/async.md).
+`encode_batch` and `encode_files` release the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`) and the calling fiber yields to the reactor too. Design notes: [docs/how-to/run-under-async.md](docs/how-to/run-under-async.md).
 
 ## CLI
 
@@ -138,6 +138,16 @@ gigatoken bench lib/gigatoken/encodings/cl100k_base.tiktoken README.md --pretoke
 
 Leave it off against a `.tiktoken` TOKENIZER and both commands raise `Gigatoken::Error` naming the valid schemes instead of crashing; for every other TOKENIZER shape (`tokenizer.json`, a packaged name, a Hub repo id) `--pretokenizer` is accepted but ignored.
 
+## Documentation
+
+In-depth docs live under [`docs/`](docs/README.md), organized by
+[Diátaxis](https://diataxis.fr/):
+
+- **Tutorial:** [Getting started](docs/tutorials/getting-started.md)
+- **How-to:** [Load a tokenizer](docs/how-to/load-a-tokenizer.md), [Tokenize files](docs/how-to/tokenize-files.md), [Packed results](docs/how-to/use-packed-results.md), [Cache budget](docs/how-to/tune-the-cache-budget.md), [Async](docs/how-to/run-under-async.md), [Measure](docs/how-to/measure-time-and-allocations.md)
+- **Reference:** [`Tokenizer`](docs/reference/tokenizer.md), [`PackedResult`](docs/reference/packed-result.md), [File sources](docs/reference/file-sources.md), [Encodings and settings](docs/reference/encodings-and-settings.md), [CLI](docs/reference/cli.md)
+- **Explanation:** [Benchmarks](docs/explanation/benchmarks.md), [Allocations](docs/explanation/allocations.md), [Async design](docs/explanation/async-design.md)
+
 ## Development
 
 ```bash
@@ -145,6 +155,7 @@ bundle install
 bundle exec rake compile    # native extension (Rust nightly, via rust-toolchain.toml)
 bundle exec rspec
 bundle exec standardrb
+ruby -Ilib bench/operations.rb   # every operation: i/s, objects and malloc per call
 ```
 
 The Ruby layer is fiber-first throughout — no `Thread`, no `Mutex`; all parallelism lives in the core's rayon pool. CI runs ubuntu + macos × Ruby 3.3/3.4/4.0, and `release.yml` cross-builds the precompiled native gems (arm64-darwin, x86_64-linux, aarch64-linux).

data/ext/gigatoken/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "gigatoken-rb"
-version = "0.2.2"
+version = "0.3.0"
 edition = "2021"
 
 [lib]

data/ext/gigatoken/src/gvl.rs

@@ -17,7 +17,7 @@
 //! (`~/architect/src/github.com/socketry/io-event/ext/io/event/worker_pool.c:309-316`).
 //! Without such a scheduler (or with one lacking a worker pool), `rb_nogvl`
 //! degrades to exactly today's behavior: release the GVL, block this thread.
-//! See `docs/rb/async-design.md` and `docs/rb/async.md` for the full design
+//! See `docs/explanation/async-design.md` and `docs/how-to/run-under-async.md` for the full design
 //! and gotchas (worker pool is opt-in, defaults to one background worker).
 //!
 //! `func`/`data1` may now run on a different OS thread than the caller (the

data/ext/gigatoken/src/tokenizer.rs

@@ -43,7 +43,7 @@ pub(crate) fn ragged_result(ruby: &Ruby, flat: Vec<u32>, lens: Vec<i64>) -> Resu
     let mut offset = 0usize;
     for len in lens {
         let len = len as usize;
-        result.push(flat[offset..offset + len].to_vec())?;
+        result.push(ruby.ary_from_iter(flat[offset..offset + len].iter().copied()))?;
         offset += len;
     }
     Ok(result)
@@ -460,7 +460,7 @@ impl BPETokenizer {
     /// Before you re-inline this "to simplify": rerun the evidence rather than
     /// trusting a number. `ruby -Ilib bench/encode_ab.rb` with the attributes
     /// stripped and again with them restored, and read
-    /// `docs/rb/benchmarks.md` first — no size resolves this on the hardware
+    /// `docs/explanation/benchmarks.md` first — no size resolves this on the hardware
     /// measured so far. The instrument is honest (an interleaved same-build
     /// run never calls a size faster or slower, at any size) and has power to
     /// catch a couple-percent effect reliably, but the attributes' real

data/lib/gigatoken/hub.rb

@@ -99,6 +99,12 @@ module Gigatoken
         revision.match?(/\A[0-9a-f]{40}\z/)
       end
 
+      # The Hub endpoint, resolved like huggingface_hub does it: HF_ENDPOINT,
+      # then https://huggingface.co.
+      def default_endpoint
+        env("HF_ENDPOINT") || DEFAULT_ENDPOINT
+      end
+
       private
 
       def env(key)
@@ -125,8 +131,9 @@ module Gigatoken
 
     # @parameter endpoint [String] the Hub endpoint to fetch from — override
     #   for pointing at a local server in tests (dependency injection, not a
-    #   mock).
-    def initialize(endpoint: DEFAULT_ENDPOINT)
+    #   mock); defaults to Hub.default_endpoint (HF_ENDPOINT, then
+    #   huggingface.co).
+    def initialize(endpoint: self.class.default_endpoint)
       @endpoint = endpoint.chomp("/")
       @internet = Async::HTTP::Internet.new
     end

data/lib/gigatoken/packed_result.rb

@@ -28,9 +28,13 @@ module Gigatoken
       lens.sum
     end
 
-    # Array of token ids for document `i`, materialized on demand.
+    # Array of token ids for document `i`, materialized on demand; negative
+    # indices count from the end and out-of-range ones give nil, like Array.
     def [](i)
-      buffer.get_values(Array.new(lens[i], :u32), @offsets[i] * 4)
+      i += size if i.negative?
+      return unless i >= 0 && i < size
+
+      buffer.values(:u32, @offsets[i] * 4, lens[i])
     end
 
     def each
@@ -42,7 +46,7 @@ module Gigatoken
     # A ragged Array of Arrays, one per document — the same shape
     # `encode_batch`/`encode_files` return with `packed: false`.
     def to_a
-      each.to_a
+      Array.new(size) { |i| self[i] }
     end
   end
 end

data/lib/gigatoken/tokenizer.rb

@@ -49,8 +49,8 @@ module Gigatoken
 
     # Load tokenizer.json from HuggingFace Hub repo `repo_id` at `revision`
     # (downloaded directly; huggingface_hub is not required).
-    def self.from_hub(repo_id, revision: "main", hub: Hub.new)
-      from_file(hub.hub_file(repo_id, "tokenizer.json", revision: revision))
+    def self.from_hub(repo_id, revision: "main", hub: nil)
+      from_file((hub || Hub.new).hub_file(repo_id, "tokenizer.json", revision: revision))
     end
 
     # Load from any of the supported source shapes: an existing file or
@@ -67,7 +67,7 @@ module Gigatoken
     # here too, raising the same explanation from_encoding gives rather than
     # reaching the Hub — but only those; an unrecognized bare name like
     # "gpt2" still dispatches to the Hub.
-    def self.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: Hub.new)
+    def self.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: nil)
       source = source.to_s
       if source.end_with?(".tiktoken")
         unless pretokenizer
@@ -83,8 +83,12 @@ module Gigatoken
       raise Error, "#{source.inspect}: no such file or directory, not a .tiktoken path, and doesn't look like a HuggingFace Hub repo id"
     end
 
+    # `data` is UTF-8 JSON whatever its encoding tag says (File.binread tags
+    # binary; a US-ASCII default_external tags that), so retag rather than let
+    # JSON.parse transcode from the tag; a UTF-8-tagged String needs no copy.
     def self.special_tokens_from_json(data)
-      added = JSON.parse(data.dup.force_encoding(Encoding::UTF_8))["added_tokens"] || []
+      data = data.dup.force_encoding(Encoding::UTF_8) unless data.encoding == Encoding::UTF_8
+      added = JSON.parse(data)["added_tokens"] || []
       added.each_with_object({}) { |t, h| h[t["content"]] = t["id"] if t["special"] }
     end
     private_class_method :special_tokens_from_json

data/lib/gigatoken/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Gigatoken
-  VERSION = "0.2.2"
+  VERSION = "0.3.0"
 end

data/rust-toolchain.toml

@@ -1,5 +1,5 @@
 [toolchain]
-channel = "nightly"
+channel = "nightly-2026-09-18"
 # Cross triples for the precompiled-gem builds (ruby-gem.yml). rustup's
 # toolchain-file auto-install is the only thing that provisions nightly
 # inside the rb-sys-dock containers, so the targets must be declared here —

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gigatoken
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.0
 platform: ruby
 authors:
 - Eric Jacobs