static_embeddings 0.1.2 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 250b847231ef37114e5c1c6b12076565eaa42d6193c305b0b107b424b12b3397
-  data.tar.gz: e59f63626b715759a286c25949b3c03bed78bc58744085716b96cd96a8fa8f98
+  metadata.gz: f72b45c2d6fa5310e3b6d5a9b0a86ab0430fc4af71f6811f9aa02655de6272a8
+  data.tar.gz: 51f0bb96a3b0adc5f2799e1df425cf1747ca521d92ae90afba59e0014f1d4106
 SHA512:
-  metadata.gz: 71f7a2e68300934fe1362c5b8f546312ced1424dc0616a752400d8d48c9d80a1b82e053592bcf882e24f3a0ba7a10044b7b17a23a2bb5ea6bbe6f8dfcb896e97
-  data.tar.gz: 957e006602a759274873d894cbaf40a50018af531a0a9cd39806ef5eea8f6159b59b4ebba3e4bc2f8daaf3fe6b89f3ed4eeb9e10d28a62eef1057a6e363226c1
+  metadata.gz: 53630b742cd2723656b5d826a82f57042dcaab1e3259758aa6a71eac20ba387a23b3fb3905f0f36f3ce641b122229f366c9af1f029205b44cd82110f538a49cb
+  data.tar.gz: 199cd93e9898b7457d5e19b9cf1857b4d78ffc46777ca26778b094fa70ee02559dc90733e4e815eba74297ef9c066940792cef09e022801ef48c704e868fa8db

data/CHANGELOG.md

@@ -1,5 +1,166 @@
 # Changelog
 
+## 0.1.4 (unreleased)
+
+Hot-path work in the C runtime. Token ids, the pooling contract and f16
+rounding are unchanged: the ASCII parity sweep and the differential fuzz
+digest against `StaticEmbeddings::Reference` still match. The
+`model2vec.StaticModel` oracle was re-run for 0.1.4 against the same
+`potion-retrieval-32m` snapshot and `.semb` as 0.1.3: 31/31 id rows, 31/31
+vectors, `min_cosine=0.9999999999989528`, `max_abs_all=2.980232238769531e-07`
+— the same printed numbers as 0.1.3. L2 still accumulates sum-of-squares in
+double; 0.1.4 does that with SIMD pairwise adds.
+
+Measured on an M1 Pro, `potion-retrieval-32m`, `samples/run_all.sh`
+`DURATION=25`, production build (no alloc-stats). The 0.1.3 comparison run
+had `STATIC_EMBEDDINGS_ALLOC_STATS=1`, which the docs cost at about 3% on
+`embed` and 13% on `tokenize`; differences smaller than that, or inside the
+usual 5–10% laptop spread, are not claimed.
+
+### Changed
+
+- **L2 normalisation is a SIMD double reduction plus one scale into `out`.**
+  It used to `scale_copy` with `1.0` and then walk the vector twice in scalar
+  double. On this model `dim=512` and short in-vocabulary English, that L2
+  slot was about 19% of an ASCII `embed_batch` sample tree. The fused path
+  has a NEON/`__aarch64__` kernel and an SSE2 kernel; the scalar tail is
+  unchanged. ASCII batch went 307k → 338k texts/s on the M1 Pro (+10%). The
+  SSE2 path is compiled, not timed.
+
+- **The AArch64 f16 top-k kernel decodes 16 halves per iteration, not 8.**
+  The x86 F16C kernel is untouched. Same 50 000-row, `k=10`, ASCII matrix:
+
+  ```text
+                       f32        f16
+  x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster   (0.1.2, F16C)
+  M1 Pro, neon-fp16  2.36 ms    1.81 ms    f16 1.3x faster   (0.1.4)
+  ```
+
+  The M1 Pro row used to read 2.29 / 2.88 ms, f16 1.3x *slower*. `docs/PERFORMANCE.md`
+  and the README follow the new numbers. On the lookup-table fallback f16 is
+  still usually slower than f32.
+
+- **Scratch buffers are reused per OS thread, and freed when the thread
+  exits.** `embed`, `tokenize`, `embed_batch` and `embed_token_ids` used to
+  `malloc`/`free` a scratch set on every call. A first cut of 0.1.4 used
+  `__thread` storage with no destructor, which leaked the heap buffers inside
+  the slot — bounded for a Puma pool, unbounded on the fiber-scheduler path
+  that `rb_thread_create`s one OS thread per large call. The slot is now a
+  `pthread_key` / `FlsAlloc` value whose destructor frees it. On release the
+  slot is trimmed back to the reserve sizes, so one `max_tokens: false`
+  document does not pin megabytes on that thread until process exit.
+  `tokenize` and the small `embed` path release the slot through `rb_ensure`,
+  so `rb_ary_push` raising cannot leave `in_use` stuck. `memory_smoke` calls
+  acquire/release so ASan/valgrind actually see this code.
+
+- **ASCII WordPiece copies an all-ASCII word as bytes** instead of
+  encoding UTF-8 through the codepoint buffer, then hashes and splits on the
+  trie with identity offsets. Output is the same ids. On the OOV tokenize
+  probes this is inside the run-to-run floor.
+
+- **Latin-1 (`U+0080..U+00FF`) lower/NFD/Mn/P/C/Zs tables are built at
+  load.** Codepoints below 256 skip the binary search into the mmap'd maps.
+  ASCII (`< 0x80`) still uses the existing 128-entry class table. Not visible
+  as texts/s on the English batch corpus.
+
+- **Vocabulary compare is an inline unaligned equality test** instead of
+  libc `memcmp`. Trie child lookup special-cases one- and two-edge nodes
+  before the linear/binary search. Same ids; no measured throughput claim.
+
+Two things were measured and not shipped. Unrolling `add_row` to 32 floats
+and stretching the prefetch distance to 8 cost about 13% on
+`random_pooling_hot_path` — that loop is already memory-bound on random 2 KiB
+row gathers. Walking the ASCII trie without `cps2` (`start += matched_len`)
+cost about 38% on the OOV tokenize probes.
+
+### Added
+
+- **Batch samples print `mean_tokens_per_text`, `unk_ratio` and
+  `tokens_per_sec`.** 307k vs 103k texts/s is not readable without token
+  density: on this corpus ASCII is 31 tokens/text and 0 unk, hashes 73 and 0,
+  unicode 68 and 4.4% unk.
+
+- **`test/scratch_tls_test.rb`.** When the instrumented allocator is loaded:
+  scratch bytes plateau across eight generations of helper threads, and a
+  large `max_tokens: false` embed must not leave the calling thread's slot
+  pinned. The `alloc_stats` CI job is what actually runs it.
+
+## 0.1.3 (unreleased)
+
+Hardening and tooling, mostly borrowed from the sibling `tg_geometry` gem after
+reading its C extension side by side with this one. No behaviour changes to
+tokenization, pooling or search: the differential fuzz digest against
+`StaticEmbeddings::Reference` is unchanged.
+
+### Fixed
+
+- **Unknown keywords are rejected instead of ignored.** `rb_hash_lookup2`
+  returns the default for an absent key, so every misspelled keyword used to be
+  silently dropped and the caller got a plausible wrong answer. Two cases were
+  actively dangerous: `embed(text, fromat: :f16)` returned a full-width f32
+  vector, and `embed_batch(texts, threds: 4)` slipped past the guard whose only
+  job is to say that `threads:` buys nothing. All eight keyword-accepting
+  methods now validate against a whitelist and name the accepted keywords in the
+  error.
+
+  ```ruby
+  model.embed("hi", fromat: :f16)
+  # ArgumentError: unknown keyword: :fromat
+  #   (accepted: :max_tokens, :format, :validate_encoding, :threads)
+  ```
+
+- **`memsize` no longer charges the mapped model to the Ruby heap.** A mapped
+  `.semb` is file-backed, shared between forked workers and reclaimable by the
+  kernel, but `ObjectSpace.memsize_of(model)` reported its full size, which fed
+  GC heuristics memory the VM could neither free nor account for. It now reports
+  only process-owned bytes; `mapped_bytes` still reports the mapping. The
+  Windows heap fallback really is process memory and is still counted.
+
+### Added
+
+- **`benchmark/` with a shared harness.** Iteration counts are calibrated so
+  every measurement runs for at least `BENCH_MIN_SECONDS`, repeats are taken,
+  the median is reported, and every row carries `spread_pct` — the best-to-worst
+  distance across repeats. This release exists partly because a 6% throughput
+  claim in 0.1.2 turned out to be inside that spread. `BENCH_FORMAT=kv` gives
+  machine-readable output. Run with `rake benchmark`. Inputs are
+  `valid_encoding?`-primed, so rows report `coderange=cached` and measure the
+  hot path rather than a string fresh from a file or a database.
+
+- **`benchmark/gvl_threshold.rb`.** `SE_GVL_UNLOCK_THRESHOLD` has never been
+  justified by data. The sweep measures both halves of the trade: on one x86-64
+  run the step at 2048 bytes cost about 32% latency and four extra allocations,
+  and it is also the point where a second Ruby thread starts making progress at
+  all. The constant is unchanged; now there is something to argue from.
+
+- **`benchmark/core_paths.rb`,** covering tokenize, single embed, batch embed,
+  pooling in isolation and top-k in both formats. It reports single-text embed
+  twice, once repeating one string and once rotating a corpus, because the gap
+  between them is the cache effect that makes hot-loop numbers unreproducible.
+
+- **`test/gc_compaction_test.rb` and a `gc_compaction` CI job.** `top_k`,
+  `embed` and `embed_batch` hand raw pointers into C and release the GVL, so
+  compaction moving Ruby objects would show up as wrong numbers rather than a
+  crash.
+
+  Getting this test to test anything took two corrections. It first used a
+  64-row matrix, which is 2 KiB against a 1 MiB `SE_TOPK_GVL_UNLOCK_THRESHOLD`,
+  so the scan never released the GVL and compaction could not interleave with
+  it. The same mistake then turned out to be hiding in the embed case: 2038
+  bytes of corpus against a 2048-byte `SE_GVL_UNLOCK_THRESHOLD`, ten bytes
+  under. Measured with a spinning neighbour thread, both old cases produced zero
+  ticks during the call and both fixed ones produce hundreds of millions. Every
+  concurrency case now asserts it is over the relevant threshold, because
+  without that assertion the test silently stops testing the moment a constant
+  or a fixture changes.
+
+- **`docs/LIMITATIONS.md`.** One page answering "does it do X", split into
+  decisions and unfinished work: no ANN index, no batched queries, no candidate
+  filtering, no int8, no Ractor, no internal parallelism, and so on.
+
+- **`docs/BENCHMARKING.md`,** covering how to read `spread_pct` and the four
+  measurement traps this repository has already fallen into.
+
 ## 0.1.2 (unreleased)
 
 A tokenizer correctness fix, the ASCII fast path it made safe to write, and an

data/README.md

@@ -22,7 +22,8 @@ batch = model.embed_batch([
 ```
 
 New to the gem? Start with `GET_STARTED.md`. Design rationale is in
-`docs/ARCHITECTURE.md`.
+`docs/ARCHITECTURE.md`, and `docs/LIMITATIONS.md` is the short answer to "does
+it do X".
 
 ## Runtime contract
 
@@ -123,15 +124,19 @@ faster path for ordinary text.
 
 ### `format: :f16`
 
+Unknown keywords raise. `embed(text, fromat: :f16)` is an `ArgumentError`
+naming the accepted keywords, not a silent f32 vector.
+
 `format:` selects the returned storage encoding only. The model always computes
 in float32. For a 512-dimensional model one vector goes from 2048 to 1024 bytes.
 It is accepted by `embed`, `embed_batch`, `embed_with_stats`, `embed_token_ids`,
 `embed_token_ids_with_stats`, `pack`, `unpack`, and the top-k helpers.
 
-Choose `f16` for the RAM and storage it saves, not for speed. It halves the
-bytes a top-k scan streams, but every row still has to be decoded before
-scoring, and which effect wins is a property of the machine — this repository's
-own samples show `f16` winning on x86-64 F16C and losing on M1 Pro. See
+Choose `f16` for the RAM and storage it saves; speed is a property of the
+machine and of the decode kernel. It halves the bytes a top-k scan streams, but
+every row still has to be decoded before scoring — this repository's own
+samples show `f16` winning on x86-64 F16C and, as of 0.1.4, on M1 Pro neon-fp16
+as well. On the lookup-table fallback it usually loses. See
 `docs/PERFORMANCE.md` before assuming either.
 
 `StaticEmbeddings.simd_backend` reports the live kernel: `"neon-fp16"`,
@@ -300,6 +305,8 @@ a test fixture and API demo, not a retrieval quality baseline.
 
 ```bash
 ruby tools/benchmark.rb                  # normalised performance budget
+bundle exec rake benchmark               # benchmark/, see docs/BENCHMARKING.md
+bundle exec rake gc_compaction           # GC.compact hardening
 bundle exec rake cancellation_timing     # timing-sensitive, excluded from rake test
 samples/run_all.sh                       # native hot-path probes; see samples/README.md
 ```
@@ -317,7 +324,7 @@ cc -O2 -std=c99 -Wall -Wextra -Iext/static_embeddings \
   ext/static_embeddings/se_unicode.c \
   ext/static_embeddings/se_tokenizer.c \
   ext/static_embeddings/se_embed.c \
-  -lm -o tmp/memory_smoke
+  -lm -pthread -o tmp/memory_smoke
 ./tmp/memory_smoke tmp/test-tiny.semb
 ```
 

data/Rakefile

@@ -0,0 +1,95 @@
+require "bundler/gem_tasks" rescue nil
+require "rake/testtask"
+
+begin
+  require "rake/extensiontask"
+  Rake::ExtensionTask.new("static_embeddings") do |ext|
+    ext.lib_dir = "lib/static_embeddings"
+    ext.ext_dir = "ext/static_embeddings"
+  end
+rescue LoadError
+  desc "Compile the extension without rake-compiler"
+  task :compile do
+    ext = File.expand_path("ext/static_embeddings", __dir__)
+    lib = File.expand_path("lib/static_embeddings", __dir__)
+    build = File.expand_path("tmp/build", __dir__)
+    FileUtils.mkdir_p(build)
+    Dir.chdir(build) do
+      sh "ruby #{ext}/extconf.rb --with-cflags=-I#{ext}"
+      sh "make -j4"
+    end
+    FileUtils.cp(File.join(build, "static_embeddings.so"), lib)
+  end
+end
+
+desc "Regenerate the synthetic source model used by the test suite"
+task :fixtures do
+  ruby "tools/make_fixture_model.rb"
+end
+
+desc "Build the demo .semb shipped inside the gem"
+task demo_model: %i[fixtures compile] do
+  ruby "tools/build_demo_model.rb"
+end
+
+desc "Check the C runtime against the Python model2vec.StaticModel oracle"
+task parity: :compile do
+  model = ENV["MODEL"] || File.expand_path("~/.cache/static_embeddings/models/potion-retrieval-32m.semb")
+  oracle = ENV["ORACLE"] || "tmp/model2vec_oracle.json"
+
+  unless File.file?(oracle)
+    abort <<~MSG
+      oracle not found: #{oracle}
+
+      Generate it first (needs Python and the model2vec package):
+
+        python3 -m venv .venv-model2vec && . .venv-model2vec/bin/activate
+        pip install -U model2vec numpy
+        python tools/model2vec_oracle.py /path/to/source-model --out #{oracle}
+    MSG
+  end
+  abort "model not found: #{model}" unless File.file?(model)
+
+  ruby "-Ilib tools/check_model2vec_parity.rb --model #{model.inspect} --oracle #{oracle.inspect}"
+end
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = FileList["test/**/*_test.rb"].exclude("test/cancellation_timing_test.rb")
+  t.warning = false
+end
+
+Rake::TestTask.new(:gc_compaction) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = ["test/gc_compaction_test.rb"]
+  t.warning = false
+end
+
+Rake::TestTask.new(:cancellation_timing) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = ["test/cancellation_timing_test.rb"]
+  t.warning = false
+end
+
+task test: %i[compile fixtures]
+task cancellation_timing: %i[compile fixtures]
+task gc_compaction: %i[compile fixtures]
+
+namespace :benchmark do
+  Dir[File.join(__dir__, "benchmark", "*.rb")].sort.each do |path|
+    name = File.basename(path, ".rb")
+    next if name.start_with?("_")
+
+    desc "Run benchmark/#{name}.rb (BENCH_MODEL=... to pick a model)"
+    task name => %i[compile fixtures] do
+      ruby path
+    end
+  end
+end
+
+desc "Run every benchmark"
+task benchmark: Dir[File.join(__dir__, "benchmark", "*.rb")]
+  .sort.map { |p| File.basename(p, ".rb") }
+  .reject { |n| n.start_with?("_") }
+  .map { |n| "benchmark:#{n}" }
+task default: %i[compile fixtures test]

data/benchmark/_support.rb

@@ -0,0 +1,196 @@
+# frozen_string_literal: true
+
+require "rbconfig"
+
+ROOT = File.expand_path("..", __dir__)
+$LOAD_PATH.unshift(File.join(ROOT, "lib")) unless $LOAD_PATH.include?(File.join(ROOT, "lib"))
+$stdout.sync = true
+$stderr.sync = true
+
+begin
+  require "static_embeddings"
+rescue LoadError
+  warn "Native extension is not built. Run: bundle exec rake compile"
+  raise
+end
+
+module SEBench
+  module_function
+
+  DEFAULT_REPEATS = 7
+  DEFAULT_MIN_SECONDS = 0.25
+  DEFAULT_WARMUP_SECONDS = 0.05
+  MAX_ITERATIONS = 100_000_000
+  CANDIDATE_WORDS = %w[
+    hello world ruby gem static embedding vector search text token
+    the a of and for local fast model cafe naive
+    postgres pipeline database query batch matrix cosine kernel window prefix runtime memory latency
+  ].freeze
+  CANDIDATE_SUFFIXES = (0..15).map(&:to_s).freeze
+
+  def env_integer(name, default, min: 1)
+    value = Integer(ENV.fetch(name, default.to_s))
+    raise ArgumentError, "#{name} must be >= #{min}, got #{value}" if value < min
+
+    value
+  end
+
+  def env_float(name, default, min: 0.0)
+    value = Float(ENV.fetch(name, default.to_s))
+    raise ArgumentError, "#{name} must be >= #{min}, got #{value}" if value < min
+
+    value
+  end
+
+  def repeats = env_integer("BENCH_REPEATS", DEFAULT_REPEATS)
+  def min_seconds = env_float("BENCH_MIN_SECONDS", DEFAULT_MIN_SECONDS)
+  def warmup_seconds = env_float("BENCH_WARMUP_SECONDS", DEFAULT_WARMUP_SECONDS)
+  def adaptive? = ENV.fetch("BENCH_ADAPTIVE", "1") != "0"
+  def kv? = ENV["BENCH_FORMAT"] == "kv"
+  def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+  def realtime
+    started = monotonic
+    yield
+    monotonic - started
+  end
+
+  def median(values)
+    sorted = values.sort
+    return 0.0 if sorted.empty?
+
+    mid = sorted.length / 2
+    sorted.length.odd? ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2.0
+  end
+
+  def model
+    @model ||= begin
+      path = ENV["BENCH_MODEL"]
+      m =
+        if path
+          StaticEmbeddings.load(path)
+        elsif StaticEmbeddings.builtin_available?
+          StaticEmbeddings.load_builtin
+        else
+          abort "No model. Set BENCH_MODEL=/path/to/model.semb or run: bundle exec rake demo_model"
+        end
+      m.warmup!
+      m
+    end
+  end
+
+  def candidate_pool
+    @candidate_pool ||= begin
+      suffixed = CANDIDATE_WORDS.flat_map do |word|
+        CANDIDATE_SUFFIXES.map { |suffix| "#{word}#{suffix}" }
+      end
+      (CANDIDATE_WORDS + suffixed).uniq.freeze
+    end
+  end
+
+  def nonzero_vector?(blob)
+    blob.unpack("e*").any? { |value| value != 0.0 }
+  end
+
+  def in_vocabulary?(word)
+    stats = model.embed_with_stats(word)
+    stats[:token_count].positive? && stats[:unk_count].zero? && nonzero_vector?(stats[:vector])
+  end
+
+  def words
+    @words ||= begin
+      pool = candidate_pool.select { |word| in_vocabulary?(word) }
+      if pool.empty?
+        abort "No benchmark candidate word is in this model's vocabulary with a non-zero vector. " \
+              "Set BENCH_MODEL to a larger model or extend SEBench::CANDIDATE_WORDS."
+      end
+      pool.freeze
+    end
+  end
+
+  def text(word_count, rng)
+    Array.new(word_count) { words[rng.rand(words.length)] }.join(" ")
+  end
+
+  def corpus(count, word_count, seed: 20_260_829)
+    rng = Random.new(seed)
+    Array.new(count) { text(word_count, rng) }.each(&:valid_encoding?).freeze
+  end
+
+  def calibrate(initial, &block)
+    iterations = initial
+    return iterations unless adaptive?
+    return iterations if min_seconds <= 0.0
+
+    loop do
+      GC.start
+      elapsed = realtime { block.call(iterations) }
+      break iterations if elapsed >= min_seconds || iterations >= MAX_ITERATIONS
+
+      scale = elapsed <= 0.0 ? 10 : (min_seconds / elapsed * 1.5).ceil
+      iterations = [[iterations * scale, iterations * 2].max, MAX_ITERATIONS].min
+    end
+  end
+
+  def measure(initial_iterations: 1, operations_per_iteration: 1, &block)
+    iterations = calibrate(initial_iterations, &block)
+
+    deadline = monotonic + warmup_seconds
+    block.call(iterations) while monotonic < deadline
+
+    samples = Array.new(repeats) do
+      GC.start
+      before = GC.stat(:total_allocated_objects)
+      seconds = realtime { block.call(iterations) }
+      [seconds, GC.stat(:total_allocated_objects) - before]
+    end
+
+    times = samples.map(&:first)
+    best = times.min
+    worst = times.max
+    operations = iterations * operations_per_iteration
+
+    {
+      iterations: iterations,
+      repeats: repeats,
+      operations: operations,
+      median_sec: median(times),
+      best_sec: best,
+      worst_sec: worst,
+      spread_pct: best.positive? ? ((worst - best) / best * 100.0) : 0.0,
+      ops_per_sec: operations / median(times),
+      ns_per_op: median(times) / operations * 1e9,
+      allocations_per_op: median(samples.map(&:last)) / operations.to_f
+    }
+  end
+
+  def header(title)
+    m = model
+    puts
+    puts "== #{title} =="
+    puts "ruby=#{RUBY_VERSION} platform=#{RUBY_PLATFORM} backend=#{StaticEmbeddings.simd_backend}"
+    puts "model=#{m.model_id} dim=#{m.dim} vocab=#{m.vocab_size} max_tokens=#{m.max_tokens}"
+    puts "repeats=#{repeats} min_seconds=#{min_seconds} adaptive=#{adaptive? ? "yes" : "no"}"
+    puts "corpus_vocabulary=#{words.length} distinct in-vocabulary words " \
+         "(small pools keep matrix rows cached and overstate throughput)"
+    puts "input_coderange=cached unless a row says otherwise"
+    puts "spread_pct is best-to-worst across repeats; treat smaller differences as noise."
+    puts
+    return if kv?
+
+    printf("%-34s %12s %12s %10s %10s\n", "case", "ns/op", "ops/sec", "spread%", "alloc/op")
+  end
+
+  def report(name, stats, **fields)
+    if kv?
+      payload = { benchmark: name, ruby: RUBY_VERSION, platform: RUBY_PLATFORM,
+                  backend: StaticEmbeddings.simd_backend, dim: model.dim }
+                .merge(fields).merge(stats)
+      puts payload.map { |k, v| "#{k}=#{v.is_a?(Float) ? format("%.6g", v) : v}" }.join(" ")
+    else
+      label = fields.empty? ? name : "#{name} #{fields.map { |k, v| "#{k}=#{v}" }.join(" ")}"
+      printf("%-34s %12.1f %12.0f %9.1f%% %10.2f\n", label[0, 34], stats[:ns_per_op],
+             stats[:ops_per_sec], stats[:spread_pct], stats[:allocations_per_op])
+    end
+  end
+end

data/benchmark/core_paths.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+require_relative "_support"
+
+SEBench.header("core_paths")
+
+model = SEBench.model
+short = SEBench.corpus(2_000, 20)
+one = short.first
+ids = model.tokenize(one)
+
+SEBench.report("tokenize", SEBench.measure(initial_iterations: 500) { |n| n.times { model.tokenize(one) } },
+               words: 20, coderange: "cached")
+
+SEBench.report("embed", SEBench.measure(initial_iterations: 500) { |n| n.times { model.embed(one) } },
+               words: 20, coderange: "cached")
+
+SEBench.report("embed rotating corpus",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { short.each { |t| model.embed(t) } }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_batch",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { model.embed_batch(short) }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_batch f16",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { model.embed_batch(short, format: :f16) }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_token_ids",
+               SEBench.measure(initial_iterations: 500) { |n| n.times { model.embed_token_ids(ids) } },
+               tokens: ids.length)
+
+query = model.embed(one)
+abort "Benchmark query pooled to a zero vector; cosine_top_k requires a non-zero query" unless SEBench.nonzero_vector?(query)
+matrix = model.embed_batch(short).freeze
+rows = matrix.bytesize / (model.dim * 4)
+
+SEBench.report("dot_top_k", SEBench.measure(initial_iterations: 5) { |n| n.times { model.dot_top_k(query, matrix, 10) } },
+               rows: rows)
+SEBench.report("cosine_top_k", SEBench.measure(initial_iterations: 5) { |n| n.times { model.cosine_top_k(query, matrix, 10) } },
+               rows: rows)
+
+query16 = model.embed(one, format: :f16)
+matrix16 = model.embed_batch(short, format: :f16).freeze
+SEBench.report("dot_top_k f16",
+               SEBench.measure(initial_iterations: 5) { |n|
+                 n.times { StaticEmbeddings.dot_top_k(query16, matrix16, 10, dim: model.dim, format: :f16) }
+               }, rows: rows)
+
+large = (SEBench.corpus(1, 200_000).first).freeze
+large.valid_encoding?
+SEBench.report("embed large truncated",
+               SEBench.measure(initial_iterations: 20) { |n| n.times { model.embed(large) } },
+               logical_bytes: large.bytesize, coderange: "cached")

data/benchmark/gvl_threshold.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+require_relative "_support"
+
+SEBench.header("gvl_threshold")
+
+THRESHOLD_BYTES = [512, 1_024, 2_040, 2_048, 2_056, 4_096, 8_192, 16_384, 65_536].freeze
+PROGRESS_BYTES = [1_024, 2_048, 8_192, 65_536].freeze
+PROGRESS_ITERATIONS = SEBench.env_integer("BENCH_GVL_PROGRESS_ITERATIONS", 200)
+PROGRESS_THREAD_YIELD_EVERY = 4_096
+
+def text_with_bytes(target_bytes)
+  rng = Random.new(target_bytes)
+  text = +""
+  text << SEBench.words[rng.rand(SEBench.words.length)] << " " while text.bytesize < target_bytes
+  text = text.byteslice(0, target_bytes).freeze
+  text.valid_encoding?
+  text
+end
+
+THRESHOLD_BYTES.each do |target_bytes|
+  text = text_with_bytes(target_bytes)
+  stats = SEBench.measure(initial_iterations: 200) { |n| n.times { SEBench.model.embed(text) } }
+  SEBench.report("embed", stats, bytes: text.bytesize, coderange: "cached")
+end
+
+puts
+puts "second-thread progress during repeated embed calls (higher is better for the caller's neighbours)"
+PROGRESS_BYTES.each do |target_bytes|
+  text = text_with_bytes(target_bytes)
+  counter = 0
+  running = true
+  neighbour = Thread.new do
+    while running
+      counter += 1
+      Thread.pass if (counter % PROGRESS_THREAD_YIELD_EVERY).zero?
+    end
+  end
+
+  elapsed =
+    begin
+      SEBench.realtime { PROGRESS_ITERATIONS.times { SEBench.model.embed(text) } }
+    ensure
+      running = false
+      neighbour.join
+    end
+
+  printf("%-34s %12d ticks %9.3f s\n", "bytes=#{target_bytes}", counter, elapsed)
+end

data/docs/ARCHITECTURE.md

@@ -174,10 +174,13 @@ storage/transport choice, not a different model. `embed_array` and
 
 The f32 compute path uses small explicit SIMD kernels where they keep the format
 simple: row accumulation and scaling use NEON/SSE when the compiler target
-exposes them, with scalar fallback everywhere else. `dot_top_k`/`cosine_top_k`
+exposes them, L2 sum-of-squares uses SIMD double (NEON on AArch64, SSE2 on
+x86) so the inverse stays a `1/sqrt` of a double accumulation, and there is a
+scalar fallback everywhere else. `dot_top_k`/`cosine_top_k`
 share the SIMD dot kernel for `format: :f32`; `cosine_top_k` additionally
 accumulates each row's sum of squares so it can divide by the true norms.
-`format: :f16` top-k decodes half components on the fly. What is still
+`format: :f16` top-k decodes half components on the fly (16-wide on AArch64
+NEON-FP16, 16-wide on x86 F16C). What is still
 deliberately absent is a dependency on BLAS: pooling is gathering random
 embedding rows, not a dense matrix multiply.
 
@@ -293,8 +296,13 @@ cadence in bytes so a long ASCII run is no less interruptible. The pooling loop
 checks every 256 rows, and the top-k scan every 1024 rows. `unblock_cancel` sets that flag when Ruby
 interrupts a GVL-free region.
 
-No global mutable state exists in C. The model is immutable after load and
-scratch buffers are per call.
+No global mutable state exists in C. The model is immutable after load.
+Scratch buffers are reused per OS thread via `pthread_key` / `FlsAlloc`, and
+the destructor frees them when the thread exits — including the short-lived
+threads the fiber scheduler path creates per large call. On release the slot
+is trimmed back to the reserve sizes, so a single unlimited-`max_tokens`
+document does not pin its working set until the thread dies. A nested call on
+the same thread allocates a one-off heap scratch that is freed on release.
 
 ## Where this fits in a RAG pipeline