static_embeddings 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 250b847231ef37114e5c1c6b12076565eaa42d6193c305b0b107b424b12b3397
-  data.tar.gz: e59f63626b715759a286c25949b3c03bed78bc58744085716b96cd96a8fa8f98
+  metadata.gz: 7ec20ad222235414c2e6f081a2cea5f932a7e3004cb78dcca5552c20208bd6a8
+  data.tar.gz: 80553b44c62b46625562accee74ac9bfe689400f05258e551dc9603105f64f7a
 SHA512:
-  metadata.gz: 71f7a2e68300934fe1362c5b8f546312ced1424dc0616a752400d8d48c9d80a1b82e053592bcf882e24f3a0ba7a10044b7b17a23a2bb5ea6bbe6f8dfcb896e97
-  data.tar.gz: 957e006602a759274873d894cbaf40a50018af531a0a9cd39806ef5eea8f6159b59b4ebba3e4bc2f8daaf3fe6b89f3ed4eeb9e10d28a62eef1057a6e363226c1
+  metadata.gz: 7fc51c583e8d0f05e5e6b7181767450e4fc93601180a638444cb715be0321d993454003d93bd9bd9f7154e5d678b57dc0fdeba7d5f86c81154f0b61886898101
+  data.tar.gz: 1739214fe214cbb86a81e643878a371547b36365d408f2e81e7e95c6c0f0b66c2e214c839f41744347c5015c82832f5a34410e585eb504dab4fe2d3a106ff459

data/CHANGELOG.md

@@ -1,5 +1,81 @@
 # Changelog
 
+## 0.1.3 (unreleased)
+
+Hardening and tooling, mostly borrowed from the sibling `tg_geometry` gem after
+reading its C extension side by side with this one. No behaviour changes to
+tokenization, pooling or search: the differential fuzz digest against
+`StaticEmbeddings::Reference` is unchanged.
+
+### Fixed
+
+- **Unknown keywords are rejected instead of ignored.** `rb_hash_lookup2`
+  returns the default for an absent key, so every misspelled keyword used to be
+  silently dropped and the caller got a plausible wrong answer. Two cases were
+  actively dangerous: `embed(text, fromat: :f16)` returned a full-width f32
+  vector, and `embed_batch(texts, threds: 4)` slipped past the guard whose only
+  job is to say that `threads:` buys nothing. All eight keyword-accepting
+  methods now validate against a whitelist and name the accepted keywords in the
+  error.
+
+  ```ruby
+  model.embed("hi", fromat: :f16)
+  # ArgumentError: unknown keyword: :fromat
+  #   (accepted: :max_tokens, :format, :validate_encoding, :threads)
+  ```
+
+- **`memsize` no longer charges the mapped model to the Ruby heap.** A mapped
+  `.semb` is file-backed, shared between forked workers and reclaimable by the
+  kernel, but `ObjectSpace.memsize_of(model)` reported its full size, which fed
+  GC heuristics memory the VM could neither free nor account for. It now reports
+  only process-owned bytes; `mapped_bytes` still reports the mapping. The
+  Windows heap fallback really is process memory and is still counted.
+
+### Added
+
+- **`benchmark/` with a shared harness.** Iteration counts are calibrated so
+  every measurement runs for at least `BENCH_MIN_SECONDS`, repeats are taken,
+  the median is reported, and every row carries `spread_pct` — the best-to-worst
+  distance across repeats. This release exists partly because a 6% throughput
+  claim in 0.1.2 turned out to be inside that spread. `BENCH_FORMAT=kv` gives
+  machine-readable output. Run with `rake benchmark`. Inputs are
+  `valid_encoding?`-primed, so rows report `coderange=cached` and measure the
+  hot path rather than a string fresh from a file or a database.
+
+- **`benchmark/gvl_threshold.rb`.** `SE_GVL_UNLOCK_THRESHOLD` has never been
+  justified by data. The sweep measures both halves of the trade: on one x86-64
+  run the step at 2048 bytes cost about 32% latency and four extra allocations,
+  and it is also the point where a second Ruby thread starts making progress at
+  all. The constant is unchanged; now there is something to argue from.
+
+- **`benchmark/core_paths.rb`,** covering tokenize, single embed, batch embed,
+  pooling in isolation and top-k in both formats. It reports single-text embed
+  twice, once repeating one string and once rotating a corpus, because the gap
+  between them is the cache effect that makes hot-loop numbers unreproducible.
+
+- **`test/gc_compaction_test.rb` and a `gc_compaction` CI job.** `top_k`,
+  `embed` and `embed_batch` hand raw pointers into C and release the GVL, so
+  compaction moving Ruby objects would show up as wrong numbers rather than a
+  crash.
+
+  Getting this test to test anything took two corrections. It first used a
+  64-row matrix, which is 2 KiB against a 1 MiB `SE_TOPK_GVL_UNLOCK_THRESHOLD`,
+  so the scan never released the GVL and compaction could not interleave with
+  it. The same mistake then turned out to be hiding in the embed case: 2038
+  bytes of corpus against a 2048-byte `SE_GVL_UNLOCK_THRESHOLD`, ten bytes
+  under. Measured with a spinning neighbour thread, both old cases produced zero
+  ticks during the call and both fixed ones produce hundreds of millions. Every
+  concurrency case now asserts it is over the relevant threshold, because
+  without that assertion the test silently stops testing the moment a constant
+  or a fixture changes.
+
+- **`docs/LIMITATIONS.md`.** One page answering "does it do X", split into
+  decisions and unfinished work: no ANN index, no batched queries, no candidate
+  filtering, no int8, no Ractor, no internal parallelism, and so on.
+
+- **`docs/BENCHMARKING.md`,** covering how to read `spread_pct` and the four
+  measurement traps this repository has already fallen into.
+
 ## 0.1.2 (unreleased)
 
 A tokenizer correctness fix, the ASCII fast path it made safe to write, and an

data/README.md

@@ -22,7 +22,8 @@ batch = model.embed_batch([
 ```
 
 New to the gem? Start with `GET_STARTED.md`. Design rationale is in
-`docs/ARCHITECTURE.md`.
+`docs/ARCHITECTURE.md`, and `docs/LIMITATIONS.md` is the short answer to "does
+it do X".
 
 ## Runtime contract
 
@@ -123,6 +124,9 @@ faster path for ordinary text.
 
 ### `format: :f16`
 
+Unknown keywords raise. `embed(text, fromat: :f16)` is an `ArgumentError`
+naming the accepted keywords, not a silent f32 vector.
+
 `format:` selects the returned storage encoding only. The model always computes
 in float32. For a 512-dimensional model one vector goes from 2048 to 1024 bytes.
 It is accepted by `embed`, `embed_batch`, `embed_with_stats`, `embed_token_ids`,
@@ -300,6 +304,8 @@ a test fixture and API demo, not a retrieval quality baseline.
 
 ```bash
 ruby tools/benchmark.rb                  # normalised performance budget
+bundle exec rake benchmark               # benchmark/, see docs/BENCHMARKING.md
+bundle exec rake gc_compaction           # GC.compact hardening
 bundle exec rake cancellation_timing     # timing-sensitive, excluded from rake test
 samples/run_all.sh                       # native hot-path probes; see samples/README.md
 ```

data/Rakefile

@@ -0,0 +1,95 @@
+require "bundler/gem_tasks" rescue nil
+require "rake/testtask"
+
+begin
+  require "rake/extensiontask"
+  Rake::ExtensionTask.new("static_embeddings") do |ext|
+    ext.lib_dir = "lib/static_embeddings"
+    ext.ext_dir = "ext/static_embeddings"
+  end
+rescue LoadError
+  desc "Compile the extension without rake-compiler"
+  task :compile do
+    ext = File.expand_path("ext/static_embeddings", __dir__)
+    lib = File.expand_path("lib/static_embeddings", __dir__)
+    build = File.expand_path("tmp/build", __dir__)
+    FileUtils.mkdir_p(build)
+    Dir.chdir(build) do
+      sh "ruby #{ext}/extconf.rb --with-cflags=-I#{ext}"
+      sh "make -j4"
+    end
+    FileUtils.cp(File.join(build, "static_embeddings.so"), lib)
+  end
+end
+
+desc "Regenerate the synthetic source model used by the test suite"
+task :fixtures do
+  ruby "tools/make_fixture_model.rb"
+end
+
+desc "Build the demo .semb shipped inside the gem"
+task demo_model: %i[fixtures compile] do
+  ruby "tools/build_demo_model.rb"
+end
+
+desc "Check the C runtime against the Python model2vec.StaticModel oracle"
+task parity: :compile do
+  model = ENV["MODEL"] || File.expand_path("~/.cache/static_embeddings/models/potion-retrieval-32m.semb")
+  oracle = ENV["ORACLE"] || "tmp/model2vec_oracle.json"
+
+  unless File.file?(oracle)
+    abort <<~MSG
+      oracle not found: #{oracle}
+
+      Generate it first (needs Python and the model2vec package):
+
+        python3 -m venv .venv-model2vec && . .venv-model2vec/bin/activate
+        pip install -U model2vec numpy
+        python tools/model2vec_oracle.py /path/to/source-model --out #{oracle}
+    MSG
+  end
+  abort "model not found: #{model}" unless File.file?(model)
+
+  ruby "-Ilib tools/check_model2vec_parity.rb --model #{model.inspect} --oracle #{oracle.inspect}"
+end
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = FileList["test/**/*_test.rb"].exclude("test/cancellation_timing_test.rb")
+  t.warning = false
+end
+
+Rake::TestTask.new(:gc_compaction) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = ["test/gc_compaction_test.rb"]
+  t.warning = false
+end
+
+Rake::TestTask.new(:cancellation_timing) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = ["test/cancellation_timing_test.rb"]
+  t.warning = false
+end
+
+task test: %i[compile fixtures]
+task cancellation_timing: %i[compile fixtures]
+task gc_compaction: %i[compile fixtures]
+
+namespace :benchmark do
+  Dir[File.join(__dir__, "benchmark", "*.rb")].sort.each do |path|
+    name = File.basename(path, ".rb")
+    next if name.start_with?("_")
+
+    desc "Run benchmark/#{name}.rb (BENCH_MODEL=... to pick a model)"
+    task name => %i[compile fixtures] do
+      ruby path
+    end
+  end
+end
+
+desc "Run every benchmark"
+task benchmark: Dir[File.join(__dir__, "benchmark", "*.rb")]
+  .sort.map { |p| File.basename(p, ".rb") }
+  .reject { |n| n.start_with?("_") }
+  .map { |n| "benchmark:#{n}" }
+task default: %i[compile fixtures test]

data/benchmark/_support.rb

@@ -0,0 +1,196 @@
+# frozen_string_literal: true
+
+require "rbconfig"
+
+ROOT = File.expand_path("..", __dir__)
+$LOAD_PATH.unshift(File.join(ROOT, "lib")) unless $LOAD_PATH.include?(File.join(ROOT, "lib"))
+$stdout.sync = true
+$stderr.sync = true
+
+begin
+  require "static_embeddings"
+rescue LoadError
+  warn "Native extension is not built. Run: bundle exec rake compile"
+  raise
+end
+
+module SEBench
+  module_function
+
+  DEFAULT_REPEATS = 7
+  DEFAULT_MIN_SECONDS = 0.25
+  DEFAULT_WARMUP_SECONDS = 0.05
+  MAX_ITERATIONS = 100_000_000
+  CANDIDATE_WORDS = %w[
+    hello world ruby gem static embedding vector search text token
+    the a of and for local fast model cafe naive
+    postgres pipeline database query batch matrix cosine kernel window prefix runtime memory latency
+  ].freeze
+  CANDIDATE_SUFFIXES = (0..15).map(&:to_s).freeze
+
+  def env_integer(name, default, min: 1)
+    value = Integer(ENV.fetch(name, default.to_s))
+    raise ArgumentError, "#{name} must be >= #{min}, got #{value}" if value < min
+
+    value
+  end
+
+  def env_float(name, default, min: 0.0)
+    value = Float(ENV.fetch(name, default.to_s))
+    raise ArgumentError, "#{name} must be >= #{min}, got #{value}" if value < min
+
+    value
+  end
+
+  def repeats = env_integer("BENCH_REPEATS", DEFAULT_REPEATS)
+  def min_seconds = env_float("BENCH_MIN_SECONDS", DEFAULT_MIN_SECONDS)
+  def warmup_seconds = env_float("BENCH_WARMUP_SECONDS", DEFAULT_WARMUP_SECONDS)
+  def adaptive? = ENV.fetch("BENCH_ADAPTIVE", "1") != "0"
+  def kv? = ENV["BENCH_FORMAT"] == "kv"
+  def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+  def realtime
+    started = monotonic
+    yield
+    monotonic - started
+  end
+
+  def median(values)
+    sorted = values.sort
+    return 0.0 if sorted.empty?
+
+    mid = sorted.length / 2
+    sorted.length.odd? ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2.0
+  end
+
+  def model
+    @model ||= begin
+      path = ENV["BENCH_MODEL"]
+      m =
+        if path
+          StaticEmbeddings.load(path)
+        elsif StaticEmbeddings.builtin_available?
+          StaticEmbeddings.load_builtin
+        else
+          abort "No model. Set BENCH_MODEL=/path/to/model.semb or run: bundle exec rake demo_model"
+        end
+      m.warmup!
+      m
+    end
+  end
+
+  def candidate_pool
+    @candidate_pool ||= begin
+      suffixed = CANDIDATE_WORDS.flat_map do |word|
+        CANDIDATE_SUFFIXES.map { |suffix| "#{word}#{suffix}" }
+      end
+      (CANDIDATE_WORDS + suffixed).uniq.freeze
+    end
+  end
+
+  def nonzero_vector?(blob)
+    blob.unpack("e*").any? { |value| value != 0.0 }
+  end
+
+  def in_vocabulary?(word)
+    stats = model.embed_with_stats(word)
+    stats[:token_count].positive? && stats[:unk_count].zero? && nonzero_vector?(stats[:vector])
+  end
+
+  def words
+    @words ||= begin
+      pool = candidate_pool.select { |word| in_vocabulary?(word) }
+      if pool.empty?
+        abort "No benchmark candidate word is in this model's vocabulary with a non-zero vector. " \
+              "Set BENCH_MODEL to a larger model or extend SEBench::CANDIDATE_WORDS."
+      end
+      pool.freeze
+    end
+  end
+
+  def text(word_count, rng)
+    Array.new(word_count) { words[rng.rand(words.length)] }.join(" ")
+  end
+
+  def corpus(count, word_count, seed: 20_260_829)
+    rng = Random.new(seed)
+    Array.new(count) { text(word_count, rng) }.each(&:valid_encoding?).freeze
+  end
+
+  def calibrate(initial, &block)
+    iterations = initial
+    return iterations unless adaptive?
+    return iterations if min_seconds <= 0.0
+
+    loop do
+      GC.start
+      elapsed = realtime { block.call(iterations) }
+      break iterations if elapsed >= min_seconds || iterations >= MAX_ITERATIONS
+
+      scale = elapsed <= 0.0 ? 10 : (min_seconds / elapsed * 1.5).ceil
+      iterations = [[iterations * scale, iterations * 2].max, MAX_ITERATIONS].min
+    end
+  end
+
+  def measure(initial_iterations: 1, operations_per_iteration: 1, &block)
+    iterations = calibrate(initial_iterations, &block)
+
+    deadline = monotonic + warmup_seconds
+    block.call(iterations) while monotonic < deadline
+
+    samples = Array.new(repeats) do
+      GC.start
+      before = GC.stat(:total_allocated_objects)
+      seconds = realtime { block.call(iterations) }
+      [seconds, GC.stat(:total_allocated_objects) - before]
+    end
+
+    times = samples.map(&:first)
+    best = times.min
+    worst = times.max
+    operations = iterations * operations_per_iteration
+
+    {
+      iterations: iterations,
+      repeats: repeats,
+      operations: operations,
+      median_sec: median(times),
+      best_sec: best,
+      worst_sec: worst,
+      spread_pct: best.positive? ? ((worst - best) / best * 100.0) : 0.0,
+      ops_per_sec: operations / median(times),
+      ns_per_op: median(times) / operations * 1e9,
+      allocations_per_op: median(samples.map(&:last)) / operations.to_f
+    }
+  end
+
+  def header(title)
+    m = model
+    puts
+    puts "== #{title} =="
+    puts "ruby=#{RUBY_VERSION} platform=#{RUBY_PLATFORM} backend=#{StaticEmbeddings.simd_backend}"
+    puts "model=#{m.model_id} dim=#{m.dim} vocab=#{m.vocab_size} max_tokens=#{m.max_tokens}"
+    puts "repeats=#{repeats} min_seconds=#{min_seconds} adaptive=#{adaptive? ? "yes" : "no"}"
+    puts "corpus_vocabulary=#{words.length} distinct in-vocabulary words " \
+         "(small pools keep matrix rows cached and overstate throughput)"
+    puts "input_coderange=cached unless a row says otherwise"
+    puts "spread_pct is best-to-worst across repeats; treat smaller differences as noise."
+    puts
+    return if kv?
+
+    printf("%-34s %12s %12s %10s %10s\n", "case", "ns/op", "ops/sec", "spread%", "alloc/op")
+  end
+
+  def report(name, stats, **fields)
+    if kv?
+      payload = { benchmark: name, ruby: RUBY_VERSION, platform: RUBY_PLATFORM,
+                  backend: StaticEmbeddings.simd_backend, dim: model.dim }
+                .merge(fields).merge(stats)
+      puts payload.map { |k, v| "#{k}=#{v.is_a?(Float) ? format("%.6g", v) : v}" }.join(" ")
+    else
+      label = fields.empty? ? name : "#{name} #{fields.map { |k, v| "#{k}=#{v}" }.join(" ")}"
+      printf("%-34s %12.1f %12.0f %9.1f%% %10.2f\n", label[0, 34], stats[:ns_per_op],
+             stats[:ops_per_sec], stats[:spread_pct], stats[:allocations_per_op])
+    end
+  end
+end

data/benchmark/core_paths.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+require_relative "_support"
+
+SEBench.header("core_paths")
+
+model = SEBench.model
+short = SEBench.corpus(2_000, 20)
+one = short.first
+ids = model.tokenize(one)
+
+SEBench.report("tokenize", SEBench.measure(initial_iterations: 500) { |n| n.times { model.tokenize(one) } },
+               words: 20, coderange: "cached")
+
+SEBench.report("embed", SEBench.measure(initial_iterations: 500) { |n| n.times { model.embed(one) } },
+               words: 20, coderange: "cached")
+
+SEBench.report("embed rotating corpus",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { short.each { |t| model.embed(t) } }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_batch",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { model.embed_batch(short) }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_batch f16",
+               SEBench.measure(initial_iterations: 1, operations_per_iteration: short.length) { |n|
+                 n.times { model.embed_batch(short, format: :f16) }
+               }, texts: short.length, coderange: "cached")
+
+SEBench.report("embed_token_ids",
+               SEBench.measure(initial_iterations: 500) { |n| n.times { model.embed_token_ids(ids) } },
+               tokens: ids.length)
+
+query = model.embed(one)
+abort "Benchmark query pooled to a zero vector; cosine_top_k requires a non-zero query" unless SEBench.nonzero_vector?(query)
+matrix = model.embed_batch(short).freeze
+rows = matrix.bytesize / (model.dim * 4)
+
+SEBench.report("dot_top_k", SEBench.measure(initial_iterations: 5) { |n| n.times { model.dot_top_k(query, matrix, 10) } },
+               rows: rows)
+SEBench.report("cosine_top_k", SEBench.measure(initial_iterations: 5) { |n| n.times { model.cosine_top_k(query, matrix, 10) } },
+               rows: rows)
+
+query16 = model.embed(one, format: :f16)
+matrix16 = model.embed_batch(short, format: :f16).freeze
+SEBench.report("dot_top_k f16",
+               SEBench.measure(initial_iterations: 5) { |n|
+                 n.times { StaticEmbeddings.dot_top_k(query16, matrix16, 10, dim: model.dim, format: :f16) }
+               }, rows: rows)
+
+large = (SEBench.corpus(1, 200_000).first).freeze
+large.valid_encoding?
+SEBench.report("embed large truncated",
+               SEBench.measure(initial_iterations: 20) { |n| n.times { model.embed(large) } },
+               logical_bytes: large.bytesize, coderange: "cached")

data/benchmark/gvl_threshold.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+require_relative "_support"
+
+SEBench.header("gvl_threshold")
+
+THRESHOLD_BYTES = [512, 1_024, 2_040, 2_048, 2_056, 4_096, 8_192, 16_384, 65_536].freeze
+PROGRESS_BYTES = [1_024, 2_048, 8_192, 65_536].freeze
+PROGRESS_ITERATIONS = SEBench.env_integer("BENCH_GVL_PROGRESS_ITERATIONS", 200)
+PROGRESS_THREAD_YIELD_EVERY = 4_096
+
+def text_with_bytes(target_bytes)
+  rng = Random.new(target_bytes)
+  text = +""
+  text << SEBench.words[rng.rand(SEBench.words.length)] << " " while text.bytesize < target_bytes
+  text = text.byteslice(0, target_bytes).freeze
+  text.valid_encoding?
+  text
+end
+
+THRESHOLD_BYTES.each do |target_bytes|
+  text = text_with_bytes(target_bytes)
+  stats = SEBench.measure(initial_iterations: 200) { |n| n.times { SEBench.model.embed(text) } }
+  SEBench.report("embed", stats, bytes: text.bytesize, coderange: "cached")
+end
+
+puts
+puts "second-thread progress during repeated embed calls (higher is better for the caller's neighbours)"
+PROGRESS_BYTES.each do |target_bytes|
+  text = text_with_bytes(target_bytes)
+  counter = 0
+  running = true
+  neighbour = Thread.new do
+    while running
+      counter += 1
+      Thread.pass if (counter % PROGRESS_THREAD_YIELD_EVERY).zero?
+    end
+  end
+
+  elapsed =
+    begin
+      SEBench.realtime { PROGRESS_ITERATIONS.times { SEBench.model.embed(text) } }
+    ensure
+      running = false
+      neighbour.join
+    end
+
+  printf("%-34s %12d ticks %9.3f s\n", "bytes=#{target_bytes}", counter, elapsed)
+end

data/docs/BENCHMARKING.md

@@ -0,0 +1,87 @@
+# Benchmarking
+
+Two different tools, for two different questions.
+
+`benchmark/` answers "how fast is this, and is the difference I am looking at
+real". `samples/` answers "where does the time go" by keeping a process alive
+for a profiler to attach to. Do not use one for the other's job.
+
+```bash
+bundle exec rake benchmark                 # everything
+bundle exec rake benchmark:core_paths
+bundle exec rake benchmark:gvl_threshold
+
+BENCH_MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
+  bundle exec rake benchmark:core_paths
+```
+
+Without `BENCH_MODEL` the demo model is used, which has `dim=8` and a 176-token
+vocabulary. It proves the harness runs; it says nothing about production speed.
+
+## Reading the output
+
+```text
+case                                      ns/op      ops/sec    spread%   alloc/op
+embed words=20                           8540.5       117089       3.0%       1.00
+```
+
+`spread_pct` is the distance between the fastest and slowest repeat. **A
+difference smaller than `spread_pct` is not a result.** On a laptop or a shared
+CI runner it is routinely 5-10%, which is wider than most changes worth making.
+If you want to claim an improvement, either get the spread down or show the
+change across several independent runs.
+
+`alloc/op` is Ruby object allocations per operation. It moves for different
+reasons than time does, and it is the more stable signal of the two.
+
+Generated text inputs are primed with `valid_encoding?` before timing, so these
+benchmarks measure the cached-coderange hot path. Fresh strings from disk or the
+database can pay an extra whole-string UTF-8 scan; see `docs/PERFORMANCE.md`.
+
+Iteration counts are calibrated automatically so every measurement runs for at
+least `BENCH_MIN_SECONDS`. Knobs:
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `BENCH_MODEL` | demo model | model to load |
+| `BENCH_REPEATS` | 7 | repeats per case, median reported |
+| `BENCH_MIN_SECONDS` | 0.25 | minimum duration of one repeat |
+| `BENCH_WARMUP_SECONDS` | 0.05 | warmup before timing |
+| `BENCH_ADAPTIVE` | 1 | set to 0 to use the literal iteration count |
+| `BENCH_FORMAT` | table | set to `kv` for machine-readable lines |
+
+## What the two benchmarks are for
+
+`core_paths` covers tokenize, single embed, batch embed, pooling in isolation
+and top-k in both formats. The `embed` and `embed rotating corpus` rows are
+deliberately both present: the first repeats one string and keeps its matrix
+rows in cache, the second rotates through 2000 texts and does not. The gap
+between them is the cache effect, and quoting the first as throughput is how you
+end up with a number nobody can reproduce.
+
+`gvl_threshold` sweeps input sizes around `SE_GVL_UNLOCK_THRESHOLD`, the point
+where a single `embed` stops running under the GVL and is handed to the batch
+machinery. Look for the step, not the slope. On one x86-64 run the step cost
+about 32% latency and four extra allocations, and it is also the point where a
+second Ruby thread starts making progress at all. Both halves of that trade have
+to be measured on your hardware before the constant is worth changing.
+
+## Traps this repository has already fallen into
+
+**Quoting noise.** An `embed_batch f16` change was reported as a 6% speedup; it
+was inside the run-to-run spread and the next run showed the opposite sign. The
+memory result from the same change was real and reproducible. Report the one you
+can reproduce.
+
+**Cache-resident working sets.** Embedding the same text in a loop keeps a tiny
+slice of the matrix hot. Real corpora do not.
+
+**Logical vs processed bytes.** With truncation active, a 3 MB document is
+tokenized only until `max_tokens` is reached. Any MB/s figure computed from the
+string length divides by bytes that were never read.
+
+**GC state.** `GC.disable` does not stop an in-flight incremental cycle from
+sweeping. The harness leaves GC on and reports allocations instead.
+
+See `docs/PERFORMANCE.md` for the budget itself and where the time actually
+goes.

data/docs/LIMITATIONS.md

@@ -0,0 +1,91 @@
+# Limitations
+
+`static_embeddings` is a runtime for one narrow thing: turning text into a
+vector with a converted Model2Vec / potion model, and scanning a matrix of such
+vectors. It is not a vector database, not a general embedding library, and not a
+tokenizer toolkit.
+
+This page is the answer to "does it do X". If X is here, the answer is no, and
+the entry says whether that is a decision or just unfinished work.
+
+## Models
+
+- **Only `.semb` files from this repository's converter.** Loading a
+  HuggingFace directory at runtime is not supported and will not be. The
+  converter is the only thing that reads third-party files, and it runs offline.
+- **Only the `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE, Unigram
+  and byte-level tokenizers are rejected at conversion time rather than
+  approximated.
+- **No model training, distillation or fine-tuning.** Produce the model with
+  upstream `model2vec`, then convert it.
+- **No transformer inference.** Static embeddings have no attention and no
+  context; a word contributes the same row wherever it appears. If you need
+  contextual embeddings, this is the wrong tool.
+
+## Runtime
+
+- **No internal parallelism.** `threads:` is rejected rather than ignored. Split
+  work at the job level and run more application workers.
+- **No Ractor support.** Model, and everything holding native state, are not
+  shareable; `Ractor.make_shareable` raises.
+- **No GPU, no BLAS, no external SIMD library.** The kernels are plain C with
+  compiler autovectorisation, plus a hand-written half-precision path.
+- **No streaming input.** `embed` and `embed_batch` take Ruby `String`s that are
+  already in memory.
+- **`embed_batch` peaks at roughly twice the returned blob**, because the result
+  is built in C memory and then copied into a Ruby `String`. Chunk large
+  corpora.
+- **Big-endian hosts are refused at load.** The header is decoded
+  little-endian explicitly, but the mmapped structures and the float matrix are
+  read in native order.
+
+## Search
+
+- **No index.** `dot_top_k` and `cosine_top_k` are exhaustive scans. There is no
+  ANN, no IVF, no HNSW, no clustering.
+- **No batched queries.** One query per scan; N queries stream the matrix N
+  times.
+- **No candidate filtering.** You cannot restrict a scan to a subset of rows.
+- **No precomputed row norms.** `cosine_top_k` recomputes each row's norm on
+  every query. Normalise your rows once and use `dot_top_k` instead.
+- **No persistence, no ids, no updates.** A matrix is a frozen `String`; mapping
+  rows to documents, appending, and deleting are the caller's problem. If you
+  want that, use pgvector and keep this gem for producing the vectors.
+- **No metric other than dot product and cosine.** No Euclidean, no Manhattan,
+  no Hamming.
+
+## Storage formats
+
+- **`f32` and `f16` only.** No int8, no binary quantisation.
+- **`f16` is a storage trade-off, not a speed feature.** See
+  `docs/PERFORMANCE.md`; it wins on some CPUs and loses on others.
+- **`f16` rounding is half-up, not ties-to-even.** Blobs written by this gem can
+  differ from NumPy or PyTorch by one ULP on exact halfway values.
+
+## Platforms
+
+- **Windows is built and smoke-tested by cross-compilation under wine, not on a
+  real Windows runner.** The loader uses `CreateFileMapping`, with a
+  read-into-heap fallback.
+- **No JRuby or TruffleRuby.** The gem is a CRuby C extension.
+
+## Verification
+
+- **`load` does not check the SHA-256.** Structural validation always runs, but
+  a bit flip inside the matrix is only caught by `verify: true` or
+  `StaticEmbeddings.verify`.
+- **Parity is per model and per runtime version.** A converted model is not
+  trusted until it has a record in `docs/MODEL_AUDIT.md`, and that record
+  expires when the tokenizer changes.
+
+## Known open questions
+
+These are not decisions, just work that has not been done or measured:
+
+- The `SE_GVL_UNLOCK_THRESHOLD` value has a benchmark now
+  (`rake benchmark:gvl_threshold`) but has not been retuned from it.
+- Top-k keeps its candidates in a sorted array, which degrades for large `k`; a
+  heap would be better somewhere above a few dozen.
+- `madvise(MADV_RANDOM)` and huge pages for the embedding section are untested.
+- The C allocation counters cover the runtime's own allocations only; Ruby heap,
+  fragmentation and mmap residency are outside them.

data/docs/MODEL_AUDIT.md

@@ -11,25 +11,27 @@ runtime half of it even though the file bytes are untouched.
 | runtime | parity | note |
 |---|---|---|
 | 0.1.1 | `parity OK`, recorded below | superseded |
-| 0.1.2 | **not re-run** | required before release |
+| 0.1.2 | **not re-run** | superseded before release |
+| 0.1.3 | `parity OK`, recorded below | current |
 
 0.1.2 changed `is_control()`, which changes token ids for any input containing
 `U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
 matches HuggingFace, because it is an implementation twin of the C runtime
-written in this repository. Only the oracle can, and the oracle now carries DEL
-and control-character rows it did not have when the result below was recorded —
-so `rows=28` would itself be evidence of a stale run.
+written in this repository. The 0.1.3 release candidate was checked against a
+fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
+characters, Unicode, OOV, long-word and truncation rows.
 
 Source model:
 
 - Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
 - Python package: `model2vec 0.9.0`
 - Oracle file: `tmp/model2vec_oracle.json`
-- Oracle rows in this recorded run: `28`
+- Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
-- Runtime at time of this record: `static_embeddings 0.1.1`
+- Runtime at time of this record: `static_embeddings 0.1.3`
 
 Converted `.semb`:
 
@@ -42,6 +44,9 @@ Converted `.semb`:
 Parity command:
 
 ```bash
+python tools/model2vec_oracle.py minishlab/potion-retrieval-32M \
+  --out tmp/model2vec_oracle.json
+
 bundle exec rake parity \
   MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
   ORACLE=tmp/model2vec_oracle.json
@@ -50,8 +55,8 @@ bundle exec rake parity \
 Parity result:
 
 ```text
-rows=28
-id_rows_checked=28
+rows=31
+id_rows_checked=31
 min_cosine=0.9999999999989528
 max_abs_all=2.980232238769531e-07
 token_id_failures=[]
@@ -68,7 +73,8 @@ Decision:
 - Unicode normalization behavior: pass
 - Unknown-word behavior: pass
 - Long input / truncation behavior: pass
+- DEL / control-character behavior: pass
 
-Accepted for runtime and benchmark use **under 0.1.1**. The `.semb` file is
-unchanged and its SHA256 still matches; what expired is the runtime half of the
-record.
+Accepted for runtime and benchmark use **under 0.1.3**. The `.semb` file is
+unchanged and its SHA256 still matches; the runtime half of the record was
+refreshed after the DEL fix.

data/docs/PERFORMANCE.md

@@ -72,6 +72,14 @@ for those. And the instrumentation is not free: about 13% on `tokenize`, 3% on
 `embed`, within noise on `embed_batch`. A run captured with it is not
 comparable to one captured without it.
 
+## Running the benchmarks
+
+`bundle exec rake benchmark`, or one at a time with `rake benchmark:core_paths`
+and `rake benchmark:gvl_threshold`. Generated text inputs are
+`valid_encoding?`-primed before timing, so these are cached-coderange hot-path
+numbers. Every row carries `spread_pct`, and a difference smaller than that is
+not a result. See `docs/BENCHMARKING.md`.
+
 ## Benchmark hygiene
 
 Three failure modes have bitten this repository already.

data/ext/static_embeddings/se_format.c

@@ -845,5 +845,8 @@ size_t se_model_memsize(const se_model_t *model) {
     if (!model)
         return 0;
 
+    if (model->mapped)
+        return sizeof(se_model_t);
+
     return sizeof(se_model_t) + model->map_size;
 }

data/ext/static_embeddings/static_embeddings.c

@@ -503,6 +503,51 @@ static VALUE lookup_option(VALUE opts, ID id) {
     return rb_hash_lookup2(opts, ID2SYM(id), Qundef);
 }
 
+typedef struct {
+    const ID *allowed;
+    size_t count;
+} keyword_check_t;
+
+static int reject_unknown_keyword_i(VALUE key, VALUE value, VALUE arg) {
+    const keyword_check_t *check = (const keyword_check_t *)arg;
+    (void)value;
+
+    if (!SYMBOL_P(key))
+        rb_raise(rb_eArgError, "keyword must be a Symbol");
+
+    ID id = SYM2ID(key);
+    for (size_t i = 0; i < check->count; i++) {
+        if (id == check->allowed[i])
+            return ST_CONTINUE;
+    }
+
+    VALUE names = rb_ary_new_capa((long)check->count);
+    for (size_t i = 0; i < check->count; i++)
+        rb_ary_push(names, rb_sprintf(":%s", rb_id2name(check->allowed[i])));
+
+    const char *name = rb_id2name(id);
+    rb_raise(rb_eArgError, "unknown keyword: :%s (accepted: %" PRIsVALUE ")", name ? name : "?",
+             rb_ary_join(names, rb_str_new_cstr(", ")));
+}
+
+static void check_keywords(VALUE opts, const ID *allowed, size_t count) {
+    if (NIL_P(opts))
+        return;
+    if (!RB_TYPE_P(opts, T_HASH))
+        rb_raise(rb_eArgError, "keywords must be a Hash");
+
+    keyword_check_t check;
+    check.allowed = allowed;
+    check.count = count;
+    rb_hash_foreach(opts, reject_unknown_keyword_i, (VALUE)&check);
+}
+
+#define SE_CHECK_KEYWORDS(opts, ...)                                                       \
+    do {                                                                                   \
+        const ID se_allowed_[] = {__VA_ARGS__};                                            \
+        check_keywords((opts), se_allowed_, sizeof(se_allowed_) / sizeof(se_allowed_[0])); \
+    } while (0)
+
 static void reject_parallel_threads(VALUE opts) {
     VALUE v = lookup_option(opts, id_threads);
     if (v == Qundef || v == Qnil)
@@ -867,6 +912,7 @@ static VALUE embed_batch_internal(VALUE self, VALUE texts, VALUE max_tokens_opt,
 static VALUE model_embed_batch(int argc, VALUE *argv, VALUE self) {
     VALUE texts, opts;
     rb_scan_args(argc, argv, "1:", &texts, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_format, id_validate_encoding, id_threads);
     reject_parallel_threads(opts);
 
     VALUE max_tokens = lookup_option(opts, id_max_tokens);
@@ -927,6 +973,7 @@ static VALUE embed_one_value(VALUE self, VALUE text, VALUE max_tokens_opt,
 static VALUE model_embed(int argc, VALUE *argv, VALUE self) {
     VALUE text, opts;
     rb_scan_args(argc, argv, "1:", &text, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_format, id_validate_encoding, id_threads);
     reject_parallel_threads(opts);
     return embed_one_value(self, text, lookup_option(opts, id_max_tokens),
                            resolve_vector_format(lookup_option(opts, id_format)),
@@ -937,6 +984,7 @@ static VALUE model_embed(int argc, VALUE *argv, VALUE self) {
 static VALUE model_embed_with_stats(int argc, VALUE *argv, VALUE self) {
     VALUE text, opts;
     rb_scan_args(argc, argv, "1:", &text, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_format, id_validate_encoding, id_threads);
     reject_parallel_threads(opts);
 
     se_token_stats_t stats;
@@ -956,6 +1004,7 @@ static VALUE model_embed_with_stats(int argc, VALUE *argv, VALUE self) {
 static VALUE model_tokenize(int argc, VALUE *argv, VALUE self) {
     VALUE text, opts;
     rb_scan_args(argc, argv, "1:", &text, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_validate_encoding);
 
     model_wrapper_t *w = get_model(self);
     Check_Type(text, T_STRING);
@@ -1168,6 +1217,7 @@ static VALUE embed_token_ids_value(VALUE self, VALUE ids_value, VALUE max_tokens
 static VALUE model_embed_token_ids(int argc, VALUE *argv, VALUE self) {
     VALUE ids_value, opts;
     rb_scan_args(argc, argv, "1:", &ids_value, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_format, id_threads);
     reject_parallel_threads(opts);
     return embed_token_ids_value(self, ids_value, lookup_option(opts, id_max_tokens),
                                  resolve_vector_format(lookup_option(opts, id_format)), NULL);
@@ -1176,6 +1226,7 @@ static VALUE model_embed_token_ids(int argc, VALUE *argv, VALUE self) {
 static VALUE model_embed_token_ids_with_stats(int argc, VALUE *argv, VALUE self) {
     VALUE ids_value, opts;
     rb_scan_args(argc, argv, "1:", &ids_value, &opts);
+    SE_CHECK_KEYWORDS(opts, id_max_tokens, id_format, id_threads);
     reject_parallel_threads(opts);
 
     se_token_stats_t stats;
@@ -1315,6 +1366,7 @@ static void topk_check_matrix(VALUE matrix, size_t matrix_bytes, se_vector_forma
 static VALUE top_k_impl(int argc, VALUE *argv, VALUE self, int cosine) {
     VALUE query, matrix, k_val, opts;
     rb_scan_args(argc, argv, "3:", &query, &matrix, &k_val, &opts);
+    SE_CHECK_KEYWORDS(opts, id_dim, id_format, id_allow_unfrozen);
     (void)self;
 
     Check_Type(query, T_STRING);

data/lib/static_embeddings/version.rb

@@ -1,3 +1,3 @@
 module StaticEmbeddings
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

data/static_embeddings.gemspec

@@ -25,8 +25,10 @@ Gem::Specification.new do |spec|
     "lib/models/*.semb",
     "exe/*",
     "ext/**/*.{c,h,rb}",
+    "benchmark/**/*.rb",
     "docs/**/*.md",
     "tools/*.rb",
+    "Rakefile",
     "static_embeddings.gemspec",
     "README.md",
     "CHANGELOG.md",

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: static_embeddings
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Roman Haydarov
@@ -80,7 +80,13 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- Rakefile
+- benchmark/_support.rb
+- benchmark/core_paths.rb
+- benchmark/gvl_threshold.rb
 - docs/ARCHITECTURE.md
+- docs/BENCHMARKING.md
+- docs/LIMITATIONS.md
 - docs/MODEL_AUDIT.md
 - docs/PERFORMANCE.md
 - exe/static_embeddings