static_embeddings 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1a6d1a316162408c80a3de7afce3e1bc1f6e7cce0707c58df8737c86e0748183
-  data.tar.gz: c6e117117e09740437da333ec7203b62695b53233d5f64caa8cd15cc16f82d7c
+  metadata.gz: 7ec20ad222235414c2e6f081a2cea5f932a7e3004cb78dcca5552c20208bd6a8
+  data.tar.gz: 80553b44c62b46625562accee74ac9bfe689400f05258e551dc9603105f64f7a
 SHA512:
-  metadata.gz: 274b82bff6ffbe3f650687b9e30a611c919ffeedea859d98864857d095100f0142f3ed56954b9420eddf17ab9aac1d879dcf7648539af9a852c4d4a08df150c3
-  data.tar.gz: e9f11243e8e73ea99ab215620fa024b7cf78ccae8c5e9b2bb1d06dd88e60862fadaa37b8d2fb56b2251740a474c5d556f262e827b35ec898b1dfeadac938fc3d
+  metadata.gz: 7fc51c583e8d0f05e5e6b7181767450e4fc93601180a638444cb715be0321d993454003d93bd9bd9f7154e5d678b57dc0fdeba7d5f86c81154f0b61886898101
+  data.tar.gz: 1739214fe214cbb86a81e643878a371547b36365d408f2e81e7e95c6c0f0b66c2e214c839f41744347c5015c82832f5a34410e585eb504dab4fe2d3a106ff459

data/CHANGELOG.md

@@ -1,5 +1,202 @@
 # Changelog
 
+## 0.1.3 (unreleased)
+
+Hardening and tooling, mostly borrowed from the sibling `tg_geometry` gem after
+reading its C extension side by side with this one. No behaviour changes to
+tokenization, pooling or search: the differential fuzz digest against
+`StaticEmbeddings::Reference` is unchanged.
+
+### Fixed
+
+- **Unknown keywords are rejected instead of ignored.** `rb_hash_lookup2`
+  returns the default for an absent key, so every misspelled keyword used to be
+  silently dropped and the caller got a plausible wrong answer. Two cases were
+  actively dangerous: `embed(text, fromat: :f16)` returned a full-width f32
+  vector, and `embed_batch(texts, threds: 4)` slipped past the guard whose only
+  job is to say that `threads:` buys nothing. All eight keyword-accepting
+  methods now validate against a whitelist and name the accepted keywords in the
+  error.
+
+  ```ruby
+  model.embed("hi", fromat: :f16)
+  # ArgumentError: unknown keyword: :fromat
+  #   (accepted: :max_tokens, :format, :validate_encoding, :threads)
+  ```
+
+- **`memsize` no longer charges the mapped model to the Ruby heap.** A mapped
+  `.semb` is file-backed, shared between forked workers and reclaimable by the
+  kernel, but `ObjectSpace.memsize_of(model)` reported its full size, which fed
+  GC heuristics memory the VM could neither free nor account for. It now reports
+  only process-owned bytes; `mapped_bytes` still reports the mapping. The
+  Windows heap fallback really is process memory and is still counted.
+
+### Added
+
+- **`benchmark/` with a shared harness.** Iteration counts are calibrated so
+  every measurement runs for at least `BENCH_MIN_SECONDS`, repeats are taken,
+  the median is reported, and every row carries `spread_pct` — the best-to-worst
+  distance across repeats. This release exists partly because a 6% throughput
+  claim in 0.1.2 turned out to be inside that spread. `BENCH_FORMAT=kv` gives
+  machine-readable output. Run with `rake benchmark`. Inputs are
+  `valid_encoding?`-primed, so rows report `coderange=cached` and measure the
+  hot path rather than a string fresh from a file or a database.
+
+- **`benchmark/gvl_threshold.rb`.** `SE_GVL_UNLOCK_THRESHOLD` has never been
+  justified by data. The sweep measures both halves of the trade: on one x86-64
+  run the step at 2048 bytes cost about 32% latency and four extra allocations,
+  and it is also the point where a second Ruby thread starts making progress at
+  all. The constant is unchanged; now there is something to argue from.
+
+- **`benchmark/core_paths.rb`,** covering tokenize, single embed, batch embed,
+  pooling in isolation and top-k in both formats. It reports single-text embed
+  twice, once repeating one string and once rotating a corpus, because the gap
+  between them is the cache effect that makes hot-loop numbers unreproducible.
+
+- **`test/gc_compaction_test.rb` and a `gc_compaction` CI job.** `top_k`,
+  `embed` and `embed_batch` hand raw pointers into C and release the GVL, so
+  compaction moving Ruby objects would show up as wrong numbers rather than a
+  crash.
+
+  Getting this test to test anything took two corrections. It first used a
+  64-row matrix, which is 2 KiB against a 1 MiB `SE_TOPK_GVL_UNLOCK_THRESHOLD`,
+  so the scan never released the GVL and compaction could not interleave with
+  it. The same mistake then turned out to be hiding in the embed case: 2038
+  bytes of corpus against a 2048-byte `SE_GVL_UNLOCK_THRESHOLD`, ten bytes
+  under. Measured with a spinning neighbour thread, both old cases produced zero
+  ticks during the call and both fixed ones produce hundreds of millions. Every
+  concurrency case now asserts it is over the relevant threshold, because
+  without that assertion the test silently stops testing the moment a constant
+  or a fixture changes.
+
+- **`docs/LIMITATIONS.md`.** One page answering "does it do X", split into
+  decisions and unfinished work: no ANN index, no batched queries, no candidate
+  filtering, no int8, no Ractor, no internal parallelism, and so on.
+
+- **`docs/BENCHMARKING.md`,** covering how to read `spread_pct` and the four
+  measurement traps this repository has already fallen into.
+
+## 0.1.2 (unreleased)
+
+A tokenizer correctness fix, the ASCII fast path it made safe to write, and an
+optional build that accounts for the runtime's own C heap.
+
+External parity against `model2vec.StaticModel` has **not** been re-run for this
+version. The DEL fix below changes token ids for any input containing `U+007F`,
+so the 0.1.1 audit record no longer covers the shipping runtime. See
+`docs/MODEL_AUDIT.md`; that re-run is the release blocker.
+
+### Fixed
+
+- **`U+007F DEL` was not treated as a control character.** `is_control()`
+  matched `cp < 0x20` only, so DEL survived `clean_text` and stayed a word byte
+  while the reference normalizer drops it. The effect was not one token: the
+  word carrying DEL fell out of the vocabulary and collapsed to `[UNK]`.
+  A 20 000-case differential fuzz against `StaticEmbeddings::Reference` produced
+  4 084 mismatches before the fix and 0 after; an exhaustive sweep of
+  `0x00..0x7F` shows DEL was the only divergent ASCII byte.
+
+- **`embed_batch(format: :f16)` allocated a second output buffer.** The f32
+  batch buffer was encoded into a freshly allocated half buffer, so an f16 call
+  peaked at 1.5x the f32 output and cost more transient C heap than f32 for the
+  same work. It now encodes in place. Output is byte-identical; on 5000 x 512
+  the transient peak drops from 16.4 MB to 11.3 MB, the same peak the f32 call
+  has. Throughput is unchanged: the run-to-run spread on that sample is wider
+  than any difference the change makes.
+
+### Added
+
+- **ASCII fast path in the tokenizer.** For `cp < 0x80` the normalizer chain
+  resolves the CJK, NFD, combining-mark and case-folding branches identically
+  every time, so ASCII runs now go through a 128-entry classification table and
+  reserve their codepoint buffer once per word instead of once per character.
+  Output is unchanged: the token-stream digest over 20 000 fuzz cases is
+  identical with and without it. Roughly 1.7x on `tokenize` and 1.2x on `embed`
+  for short English text, 2x on `tokenize` with `max_tokens: false`. Non-ASCII
+  input is unaffected.
+
+- **`test/ascii_parity_test.rb`.** Exhaustive `0x00..0x7F` sweep in six
+  contexts, boundary codepoints across the ASCII/Unicode edge, deterministic
+  differential fuzz against `Reference` (`FUZZ_N`, `FUZZ_SEED`), truncation
+  boundaries, and DEL walked across the prefix window so the `embed` path is
+  covered and not just `tokenize`. Four of its cases fail on 0.1.1.
+
+- **`test/validate_encoding_test.rb`** and verify coverage in
+  `test/format_test.rb`: mode agreement on valid input, invalid bytes inside
+  and past the window, cached-broken coderange, non-UTF-8 encodings, batch
+  index reporting, tamper detection, truncated and foreign files, and an
+  allocation bound proving `verify` does not hold the file in memory.
+
+- **`test/cancellation_timing_test.rb`.** Measures how far past a `Timeout`
+  deadline `embed` keeps running. It is excluded from `rake test` because it is
+  timing-sensitive; run it with `rake cancellation_timing`.
+
+- **`validate_encoding: :full | :prefix`** on `embed`, `embed_batch`,
+  `embed_with_stats` and `tokenize`. `embed` has to establish that its `String`
+  is valid UTF-8, and when Ruby has not computed the coderange yet that scan is
+  O(total bytes) and runs before the prefix window is chosen — so the
+  large-input path was not actually bounded on a freshly read document.
+  `:prefix` skips the up-front scan and lets the tokenizer validate the bytes it
+  reads: 442 µs to 78 µs on a 3 MB string, against 71 µs for the same string
+  with its coderange cached. `:full` stays the default, because `:prefix`
+  changes behaviour — invalid bytes past the truncation window are no longer
+  seen. A coderange Ruby has already computed is honoured in both modes.
+
+- **Optional C allocation accounting.** Building with
+  `STATIC_EMBEDDINGS_ALLOC_STATS=1` wraps the runtime's own allocations and
+  exposes per-category bytes and counts, which the sample harness prints as
+  `c_alloc.<category>.<metric>`. Default builds do not define the internal
+  methods and do not pay for the counters. See `docs/PERFORMANCE.md`.
+
+- **CI jobs `parity_fuzz`, `timing`, `alloc_stats` and `windows_loader`,** plus
+  a sanitizer build of the instrumented allocator inside the existing `memory`
+  job. The Windows job cross-compiles with mingw-w64 and runs the result under
+  wine, because that branch of `se_model_open` is the one path no other job
+  builds. `alloc_stats` is separate because it has to `rake clobber` first, and
+  clobber removes `tmp/`.
+
+### Changed
+
+- **The ASCII fast path checks the cancellation flag** on the same cadence as
+  the main loop. A run of ASCII bytes is consumed inside one C call, so without
+  the check a `max_tokens: false` call on a large document ignored its deadline
+  until the whole document was tokenized: 0.33 s against a 0.05 s `Timeout` on a
+  16 MB input, versus 0.05 s now.
+
+- **`ext/static_embeddings/static_embeddings.c` was split.** The f16 codec and
+  kernels moved to `se_f16.c`, the top-k kernels to `se_topk.c`, and the
+  allocation counters live in `se_alloc_stats.c`. Shared overflow-checked size
+  helpers, `static_assert`s on the mmapped struct layouts, and an explicit
+  rejection of big-endian hosts moved into `se_internal.h`.
+
+- **`Format.verify` streams the file.** It read the whole model and then
+  duplicated it to zero the checksum field, so verifying a 135 MB artifact
+  needed roughly 270 MB of transient `String` before hashing started. Peak is
+  now one 1 MiB chunk: measured on a 62 MB model, 433 ms and 280 MB of peak RSS
+  down to 249 ms and 30 MB.
+
+- **Windows maps the model instead of reading it into the heap.** The POSIX
+  loader has always used `mmap`; Windows did `fopen` plus `fread` into a
+  private allocation, so every process paid a full read before the first query
+  and a private copy of the whole model. It now uses
+  `CreateFileMapping`/`MapViewOfFile`, with the old read path kept as a
+  fallback for filesystems that refuse to map. Verified by cross-compiling with
+  mingw-w64 and running under wine against a real `.semb`: same `map_size`,
+  same vectors as the POSIX build, and `mapped=1`.
+
+- **Throughput figures separate logical from processed bytes.**
+  `tools/benchmark.rb` reports how many texts were truncated and refuses a
+  ns/byte figure when truncation makes the two differ; the large-input samples
+  print a measured `processed_input_mb_per_sec` alongside the logical one.
+
+- **`samples/run_all.sh` captures `rake test`** into `00_test.log`, so a run
+  directory can back both "samples green" and "tests green". `TEST=0` skips it.
+
+- **Documentation.** The README no longer implies `embed` is constant-time in
+  input size (the prefix window bounds the tokenizer, not the UTF-8 validity
+  scan) and no longer claims `f16` is faster, since this repository's own
+  samples show it winning on x86-64 F16C and losing on M1 Pro.
+
 ## 0.1.1 (unreleased)
 
 Safety and correctness release. Everything here was found by re-reviewing 0.1.0