gigatoken 0.2.2-aarch64-linux → 0.3.0-aarch64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4cc04817cf7e36afff107fbe7e5fd71d856c733d4afd57c5000ebd977006c6d4
-  data.tar.gz: 17ece4ff18d0d63d1d3d6d1443ee037c6bb96310e2b6abe1fa1e4bb6fbf0b421
+  metadata.gz: 281a4ef78df3d94470e35cb463085c9da8e9bb3549c552bf6ab66f3bb4df47b1
+  data.tar.gz: f3d1531a68d00ab75abf45121b6a2df3ea113a7da39946b4e480ad7e4adea338
 SHA512:
-  metadata.gz: cbeeda4b3288656179f675beb7a7dfcb998ac068be31e794c11fb3e3c80879a6f1492bcb656ca8d03b1e6084bb2b312cb68efabdd889611c7fe47c3f5558e925
-  data.tar.gz: 99aa1445f17e51eb6aa746d12d18357c3e86522925487f2c60074a833f873232ff6d38e47d8c73f320afa5337a9d19b232e1b81029d70851d36dc5d6425e563e
+  metadata.gz: fb9419f43f11adb7f0e181e5606b008f911439f486b332acc2ddbfd06e5e798721184ec6b37122e6dadc732b3543dc5666e3c53d38ad62a234948ba0c9abdc8e
+  data.tar.gz: f867d2440b5fda9a43fb67f9bd7b8159b933ee073eac1042e8176ccbbeef1ff69eefde75b6215dd8db1c909ece1dd7b6893a0d68b9c1217f2de459325024e180

data/README.md

@@ -15,7 +15,7 @@ Zero-copy Ruby bindings for [marcelroed/gigatoken](https://github.com/marcelroed
 
 Mac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization, gigatoken just does it faster. The Python row includes a fix for [marcelroed/gigatoken#38](https://github.com/marcelroed/gigatoken/issues/38) — a hidden memcpy in `shrink_to_fit()` we found while chasing the last of the Ruby–Python gap and sent upstream (one-line fix; without it the wheel lands around 7.4 GB/s).
 
-**340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/rb/benchmarks.md](docs/rb/benchmarks.md).
+**340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology & exact counts: [docs/explanation/benchmarks.md](docs/explanation/benchmarks.md).
 
 ## Install
 
@@ -119,7 +119,7 @@ packed[3]           # => document 3's ids as an Array, on demand
 
 ### Async
 
-`encode_batch` and `encode_files` release the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`) and the calling fiber yields to the reactor too. Design notes: [docs/rb/async.md](docs/rb/async.md).
+`encode_batch` and `encode_files` release the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`) and the calling fiber yields to the reactor too. Design notes: [docs/how-to/run-under-async.md](docs/how-to/run-under-async.md).
 
 ## CLI
 
@@ -138,6 +138,16 @@ gigatoken bench lib/gigatoken/encodings/cl100k_base.tiktoken README.md --pretoke
 
 Leave it off against a `.tiktoken` TOKENIZER and both commands raise `Gigatoken::Error` naming the valid schemes instead of crashing; for every other TOKENIZER shape (`tokenizer.json`, a packaged name, a Hub repo id) `--pretokenizer` is accepted but ignored.
 
+## Documentation
+
+In-depth docs live under [`docs/`](docs/README.md), organized by
+[Diátaxis](https://diataxis.fr/):
+
+- **Tutorial:** [Getting started](docs/tutorials/getting-started.md)
+- **How-to:** [Load a tokenizer](docs/how-to/load-a-tokenizer.md), [Tokenize files](docs/how-to/tokenize-files.md), [Packed results](docs/how-to/use-packed-results.md), [Cache budget](docs/how-to/tune-the-cache-budget.md), [Async](docs/how-to/run-under-async.md), [Measure](docs/how-to/measure-time-and-allocations.md)
+- **Reference:** [`Tokenizer`](docs/reference/tokenizer.md), [`PackedResult`](docs/reference/packed-result.md), [File sources](docs/reference/file-sources.md), [Encodings and settings](docs/reference/encodings-and-settings.md), [CLI](docs/reference/cli.md)
+- **Explanation:** [Benchmarks](docs/explanation/benchmarks.md), [Allocations](docs/explanation/allocations.md), [Async design](docs/explanation/async-design.md)
+
 ## Development
 
 ```bash
@@ -145,6 +155,7 @@ bundle install
 bundle exec rake compile    # native extension (Rust nightly, via rust-toolchain.toml)
 bundle exec rspec
 bundle exec standardrb
+ruby -Ilib bench/operations.rb   # every operation: i/s, objects and malloc per call
 ```
 
 The Ruby layer is fiber-first throughout — no `Thread`, no `Mutex`; all parallelism lives in the core's rayon pool. CI runs ubuntu + macos × Ruby 3.3/3.4/4.0, and `release.yml` cross-builds the precompiled native gems (arm64-darwin, x86_64-linux, aarch64-linux).

data/lib/gigatoken/hub.rb

@@ -99,6 +99,12 @@ module Gigatoken
         revision.match?(/\A[0-9a-f]{40}\z/)
       end
 
+      # The Hub endpoint, resolved like huggingface_hub does it: HF_ENDPOINT,
+      # then https://huggingface.co.
+      def default_endpoint
+        env("HF_ENDPOINT") || DEFAULT_ENDPOINT
+      end
+
       private
 
       def env(key)
@@ -125,8 +131,9 @@ module Gigatoken
 
     # @parameter endpoint [String] the Hub endpoint to fetch from — override
     #   for pointing at a local server in tests (dependency injection, not a
-    #   mock).
-    def initialize(endpoint: DEFAULT_ENDPOINT)
+    #   mock); defaults to Hub.default_endpoint (HF_ENDPOINT, then
+    #   huggingface.co).
+    def initialize(endpoint: self.class.default_endpoint)
       @endpoint = endpoint.chomp("/")
       @internet = Async::HTTP::Internet.new
     end

data/lib/gigatoken/packed_result.rb

@@ -28,9 +28,13 @@ module Gigatoken
       lens.sum
     end
 
-    # Array of token ids for document `i`, materialized on demand.
+    # Array of token ids for document `i`, materialized on demand; negative
+    # indices count from the end and out-of-range ones give nil, like Array.
     def [](i)
-      buffer.get_values(Array.new(lens[i], :u32), @offsets[i] * 4)
+      i += size if i.negative?
+      return unless i >= 0 && i < size
+
+      buffer.values(:u32, @offsets[i] * 4, lens[i])
     end
 
     def each
@@ -42,7 +46,7 @@ module Gigatoken
     # A ragged Array of Arrays, one per document — the same shape
     # `encode_batch`/`encode_files` return with `packed: false`.
     def to_a
-      each.to_a
+      Array.new(size) { |i| self[i] }
     end
   end
 end

data/lib/gigatoken/tokenizer.rb

@@ -49,8 +49,8 @@ module Gigatoken
 
     # Load tokenizer.json from HuggingFace Hub repo `repo_id` at `revision`
     # (downloaded directly; huggingface_hub is not required).
-    def self.from_hub(repo_id, revision: "main", hub: Hub.new)
-      from_file(hub.hub_file(repo_id, "tokenizer.json", revision: revision))
+    def self.from_hub(repo_id, revision: "main", hub: nil)
+      from_file((hub || Hub.new).hub_file(repo_id, "tokenizer.json", revision: revision))
     end
 
     # Load from any of the supported source shapes: an existing file or
@@ -67,7 +67,7 @@ module Gigatoken
     # here too, raising the same explanation from_encoding gives rather than
     # reaching the Hub — but only those; an unrecognized bare name like
     # "gpt2" still dispatches to the Hub.
-    def self.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: Hub.new)
+    def self.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: nil)
       source = source.to_s
       if source.end_with?(".tiktoken")
         unless pretokenizer
@@ -83,8 +83,12 @@ module Gigatoken
       raise Error, "#{source.inspect}: no such file or directory, not a .tiktoken path, and doesn't look like a HuggingFace Hub repo id"
     end
 
+    # `data` is UTF-8 JSON whatever its encoding tag says (File.binread tags
+    # binary; a US-ASCII default_external tags that), so retag rather than let
+    # JSON.parse transcode from the tag; a UTF-8-tagged String needs no copy.
     def self.special_tokens_from_json(data)
-      added = JSON.parse(data.dup.force_encoding(Encoding::UTF_8))["added_tokens"] || []
+      data = data.dup.force_encoding(Encoding::UTF_8) unless data.encoding == Encoding::UTF_8
+      added = JSON.parse(data)["added_tokens"] || []
       added.each_with_object({}) { |t, h| h[t["content"]] = t["id"] if t["special"] }
     end
     private_class_method :special_tokens_from_json

data/lib/gigatoken/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Gigatoken
-  VERSION = "0.2.2"
+  VERSION = "0.3.0"
 end

data/rust-toolchain.toml

@@ -1,5 +1,5 @@
 [toolchain]
-channel = "nightly"
+channel = "nightly-2026-09-18"
 # Cross triples for the precompiled-gem builds (ruby-gem.yml). rustup's
 # toolchain-file auto-install is the only thing that provisions nightly
 # inside the rb-sys-dock containers, so the targets must be declared here —

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gigatoken
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.0
 platform: aarch64-linux
 authors:
 - Eric Jacobs
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2026-09-08 00:00:00.000000000 Z
+date: 2026-09-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: async