static_embeddings 0.1.3 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7ec20ad222235414c2e6f081a2cea5f932a7e3004cb78dcca5552c20208bd6a8
-  data.tar.gz: 80553b44c62b46625562accee74ac9bfe689400f05258e551dc9603105f64f7a
+  metadata.gz: fb1d3530fba4fad467251429797fdcaae527b5c6c18413e8847002e0c3f3b5e4
+  data.tar.gz: f02bb4c129449803a57b08bf9b7186af08a6d21edee1e68d18178b248b855e0b
 SHA512:
-  metadata.gz: 7fc51c583e8d0f05e5e6b7181767450e4fc93601180a638444cb715be0321d993454003d93bd9bd9f7154e5d678b57dc0fdeba7d5f86c81154f0b61886898101
-  data.tar.gz: 1739214fe214cbb86a81e643878a371547b36365d408f2e81e7e95c6c0f0b66c2e214c839f41744347c5015c82832f5a34410e585eb504dab4fe2d3a106ff459
+  metadata.gz: cec0a3959cb3f093c7bb2b06213b4c7e983986466a115eb79e0b045a5b52fdda9797be41365db41a74d85bb52c384c147d68d3d84c503fef10c122bba7b72333
+  data.tar.gz: e99039606af3de0106e1ac26efaa78a06f09acbd9c2faaae9b4ca17339bdeb4830ba922bef84dc141317898556d09b1ec41cf1ebfebcce5e141d877c3fc34598

data/CHANGELOG.md

@@ -1,5 +1,158 @@
 # Changelog
 
+## 0.1.5
+
+Correctness/parity release. The native runtime architecture is unchanged, but
+the tokenizer/pooling contract is now pinned explicitly to `model2vec 0.9.0` +
+`tokenizers 0.23.1`, and CI contains an independent upstream boundary corpus
+instead of treating the Ruby `Reference` twin as proof of compatibility.
+
+### Fixed
+
+- **`[UNK]` is filtered before the embedding token cap.** `max_tokens: 512` now
+  means the first 512 usable (non-UNK, for `UNK_DROP`) ids, matching
+  `StaticModel` pooling semantics. Long OOV-heavy text can therefore scan past
+  512 raw ids instead of silently averaging fewer rows. `embed_with_stats` and
+  `embed_token_ids_with_stats` expose `pooled_count` alongside raw
+  `token_count` / `unk_count`.
+- **The supported Hugging Face AddedVocabulary subset is implemented.** The five
+  standard BERT special tokens (`[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]`)
+  are extracted literally before normalization when the source tokenizer marks
+  them `special: true, normalized: false`. Converter and runtime now share that
+  contract instead of merely whitelisting the metadata.
+- **BertNormalizer behavior matches the pinned Rust tokenizer at the known edges.**
+  The CJK Extension E range starts at `U+2B920` as in `tokenizers 0.23.1`, not
+  the Python `transformers` boundary `U+2B820`. The boundary corpus also locks
+  the actual `unicode_categories` 0.1.1 behavior for unassigned (`Cn`)
+  codepoints: despite a misleading comment in upstream `bert.rs`, that crate's
+  `is_other()` implementation checks Cc/Cf/Co, not Cn.
+- **Vocabulary hash lookup is bounded by the validated `max_probe`,** and model
+  loading rejects hash tables above the converter's 0.70 load factor.
+- **`load_model` model ids cannot escape the cache** through absolute paths,
+  `..`, empty path components, backslash traversal or NUL bytes.
+
+### Changed
+
+- **`.semb` format v3.** The header records the supported added-token mask and
+  the usable-id truncation policy. v2 files must be reconverted; this avoids a
+  runtime silently applying 0.1.5 semantics to a file whose tokenizer contract
+  was recorded by an older converter.
+- **Model2Vec's character pre-truncation heuristic is intentionally not copied.**
+  `max_tokens` means actual usable tokenizer ids. The external oracle records
+  rows where `StaticModel`'s `max_length * median_token_length` character cut
+  changes ids and reports them as an explicit semantic deviation rather than a
+  false parity failure/pass.
+- **The converter reads safetensors incrementally and accepts F32, F16 and BF16
+  embedding tensors.** `.semb` rows remain float32. `Format.write` also streams
+  sections and computes the checksum incrementally instead of assembling a
+  second whole-file Ruby string.
+- **Plain `require "static_embeddings"` loads runtime code only.** Converter,
+  safetensors, Unicode-table generation and the Ruby `Reference` are loaded only
+  by offline tooling/tests. The `static_embeddings convert` CLI explicitly loads
+  the converter before reading converter-specific defaults, so lazy loading does
+  not break the documented executable path.
+- **TLS token-id scratch keeps a bounded high-water mark.** It still reserves only
+  512 ids initially, but capacities grown by OOV-heavy usable-token truncation are
+  retained up to 8192 ids between calls. This removes grow/trim realloc churn on
+  realistic truncated inputs without pinning arbitrarily large per-thread buffers.
+
+### Verification
+
+- Added a deterministic **434-row boundary corpus** covering control/unassigned
+  boundaries, every supported CJK range edge ±1, punctuation classes, whitespace, combining
+  marks, AddedVocabulary, 511/512/513 boundaries, OOV ratios and long-prefix
+  cases.
+- CI now installs pinned `model2vec==0.9.0` / `tokenizers==0.23.1`, generates an
+  external oracle, and separately checks raw HF ids, usable ids, embedding
+  parity where the contracts are identical, and intentional character-precut
+  deviations. The existing 50k native-vs-Ruby differential fuzz remains as a
+  separate implementation-consistency layer.
+
+## 0.1.4 (unreleased)
+
+Hot-path work in the C runtime. Token ids, the pooling contract and f16
+rounding are unchanged: the ASCII parity sweep and the differential fuzz
+digest against `StaticEmbeddings::Reference` still match. The
+`model2vec.StaticModel` oracle was re-run for 0.1.4 against the same
+`potion-retrieval-32m` snapshot and `.semb` as 0.1.3: 31/31 id rows, 31/31
+vectors, `min_cosine=0.9999999999989528`, `max_abs_all=2.980232238769531e-07`
+— the same printed numbers as 0.1.3. L2 still accumulates sum-of-squares in
+double; 0.1.4 does that with SIMD pairwise adds.
+
+Measured on an M1 Pro, `potion-retrieval-32m`, `samples/run_all.sh`
+`DURATION=25`, production build (no alloc-stats). The 0.1.3 comparison run
+had `STATIC_EMBEDDINGS_ALLOC_STATS=1`, which the docs cost at about 3% on
+`embed` and 13% on `tokenize`; differences smaller than that, or inside the
+usual 5–10% laptop spread, are not claimed.
+
+### Changed
+
+- **L2 normalisation is a SIMD double reduction plus one scale into `out`.**
+  It used to `scale_copy` with `1.0` and then walk the vector twice in scalar
+  double. On this model `dim=512` and short in-vocabulary English, that L2
+  slot was about 19% of an ASCII `embed_batch` sample tree. The fused path
+  has a NEON/`__aarch64__` kernel and an SSE2 kernel; the scalar tail is
+  unchanged. ASCII batch went 307k → 338k texts/s on the M1 Pro (+10%). The
+  SSE2 path is compiled, not timed.
+
+- **The AArch64 f16 top-k kernel decodes 16 halves per iteration, not 8.**
+  The x86 F16C kernel is untouched. Same 50 000-row, `k=10`, ASCII matrix:
+
+  ```text
+                       f32        f16
+  x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster   (0.1.2, F16C)
+  M1 Pro, neon-fp16  2.36 ms    1.81 ms    f16 1.3x faster   (0.1.4)
+  ```
+
+  The M1 Pro row used to read 2.29 / 2.88 ms, f16 1.3x *slower*. `docs/PERFORMANCE.md`
+  and the README follow the new numbers. On the lookup-table fallback f16 is
+  still usually slower than f32.
+
+- **Scratch buffers are reused per OS thread, and freed when the thread
+  exits.** `embed`, `tokenize`, `embed_batch` and `embed_token_ids` used to
+  `malloc`/`free` a scratch set on every call. A first cut of 0.1.4 used
+  `__thread` storage with no destructor, which leaked the heap buffers inside
+  the slot — bounded for a Puma pool, unbounded on the fiber-scheduler path
+  that `rb_thread_create`s one OS thread per large call. The slot is now a
+  `pthread_key` / `FlsAlloc` value whose destructor frees it. On release the
+  slot is trimmed back to the reserve sizes, so one `max_tokens: false`
+  document does not pin megabytes on that thread until process exit.
+  `tokenize` and the small `embed` path release the slot through `rb_ensure`,
+  so `rb_ary_push` raising cannot leave `in_use` stuck. `memory_smoke` calls
+  acquire/release so ASan/valgrind actually see this code.
+
+- **ASCII WordPiece copies an all-ASCII word as bytes** instead of
+  encoding UTF-8 through the codepoint buffer, then hashes and splits on the
+  trie with identity offsets. Output is the same ids. On the OOV tokenize
+  probes this is inside the run-to-run floor.
+
+- **Latin-1 (`U+0080..U+00FF`) lower/NFD/Mn/P/C/Zs tables are built at
+  load.** Codepoints below 256 skip the binary search into the mmap'd maps.
+  ASCII (`< 0x80`) still uses the existing 128-entry class table. Not visible
+  as texts/s on the English batch corpus.
+
+- **Vocabulary compare is an inline unaligned equality test** instead of
+  libc `memcmp`. Trie child lookup special-cases one- and two-edge nodes
+  before the linear/binary search. Same ids; no measured throughput claim.
+
+Two things were measured and not shipped. Unrolling `add_row` to 32 floats
+and stretching the prefetch distance to 8 cost about 13% on
+`random_pooling_hot_path` — that loop is already memory-bound on random 2 KiB
+row gathers. Walking the ASCII trie without `cps2` (`start += matched_len`)
+cost about 38% on the OOV tokenize probes.
+
+### Added
+
+- **Batch samples print `mean_tokens_per_text`, `unk_ratio` and
+  `tokens_per_sec`.** 307k vs 103k texts/s is not readable without token
+  density: on this corpus ASCII is 31 tokens/text and 0 unk, hashes 73 and 0,
+  unicode 68 and 4.4% unk.
+
+- **`test/scratch_tls_test.rb`.** When the instrumented allocator is loaded:
+  scratch bytes plateau across eight generations of helper threads, and a
+  large `max_tokens: false` embed must not leave the calling thread's slot
+  pinned. The `alloc_stats` CI job is what actually runs it.
+
 ## 0.1.3 (unreleased)
 
 Hardening and tooling, mostly borrowed from the sibling `tg_geometry` gem after

data/README.md

@@ -32,8 +32,11 @@ never arbitrary HuggingFace models. A `.semb` file carries validated metadata, a
 BERT WordPiece tokenizer profile, an mmap-ready vocabulary lookup, float32
 embedding rows, and provenance plus a checksum.
 
-The only supported tokenizer profile is `BERT_WORDPIECE_V1`. The converter
-rejects unsupported tokenizer features rather than approximating them.
+The only supported tokenizer profile is `BERT_WORDPIECE_V1`, pinned for
+compatibility to `tokenizers 0.23.1`. The converter rejects unsupported
+tokenizer features rather than approximating them. `.semb` format v3 also
+records which of the five standard BERT added tokens are active. Files produced
+by 0.1.4 and earlier use format v2 and must be reconverted.
 
 ## Converting a model
 
@@ -57,6 +60,9 @@ model = StaticEmbeddings.load(ENV.fetch("EMBEDDING_MODEL"))   # explicit path
 Convert during image build or deploy preparation; production should only ever
 see `.semb` files.
 
+The offline converter accepts F32, F16 and BF16 safetensors embedding matrices;
+all three are converted to float32 rows in `.semb`.
+
 ### Verify once, not on every boot
 
 `load` does not check the SHA-256 by default: hashing a 135 MB file at boot
@@ -98,6 +104,7 @@ stats = model.embed_with_stats("postgres pipeline mode in Ruby")
 stats[:vector]
 stats[:token_count]
 stats[:unk_count]
+stats[:pooled_count]  # rows actually eligible for pooling
 stats[:truncated]
 
 ids         = model.tokenize("postgres pipeline mode in Ruby")
@@ -116,11 +123,19 @@ Output is a binary `String` of little-endian float32 values in row-major order.
 `embed_array` and `embed_batch_arrays` decode that into Ruby `Float` objects and
 exist for debugging and application code, not for the hot path.
 
-`embed_token_ids` pools ids you supply, applying the same `max_tokens`
-truncation as `embed`, so `embed_token_ids(model.tokenize(text))` equals
-`embed(text)`. Pass `max_tokens: false` to pool every id. It is for reusing a
-cached tokenization and for benchmarking pooling in isolation; it is not a
-faster path for ordinary text.
+`tokenize` exposes raw WordPiece ids, including `[UNK]`, and its own
+`max_tokens` cap is therefore a raw-id cap. Embedding has a different, upstream
+pooling rule: for `UNK_DROP`, `[UNK]` is removed first and `max_tokens` is then
+applied to usable ids. To cache tokenization without changing the embedding,
+cache it unbounded:
+
+```ruby
+ids = model.tokenize(text, max_tokens: false)
+model.embed_token_ids(ids) == model.embed(text)
+```
+
+`embed_token_ids` applies that same usable-id cap. This distinction matters only
+when `[UNK]` appears around the truncation boundary.
 
 ### `format: :f16`
 
@@ -132,10 +147,11 @@ in float32. For a 512-dimensional model one vector goes from 2048 to 1024 bytes.
 It is accepted by `embed`, `embed_batch`, `embed_with_stats`, `embed_token_ids`,
 `embed_token_ids_with_stats`, `pack`, `unpack`, and the top-k helpers.
 
-Choose `f16` for the RAM and storage it saves, not for speed. It halves the
-bytes a top-k scan streams, but every row still has to be decoded before
-scoring, and which effect wins is a property of the machine — this repository's
-own samples show `f16` winning on x86-64 F16C and losing on M1 Pro. See
+Choose `f16` for the RAM and storage it saves; speed is a property of the
+machine and of the decode kernel. It halves the bytes a top-k scan streams, but
+every row still has to be decoded before scoring — this repository's own
+samples show `f16` winning on x86-64 F16C and, as of 0.1.4, on M1 Pro neon-fp16
+as well. On the lookup-table fallback it usually loses. See
 `docs/PERFORMANCE.md` before assuming either.
 
 `StaticEmbeddings.simd_backend` reports the live kernel: `"neon-fp16"`,
@@ -174,13 +190,13 @@ model.dot_top_k(query, MATRIX, 10)          # lock-free, concurrent, GVL release
 
 ## Large inputs
 
-When truncation is active the runtime does not copy the whole string into C
-memory. It copies a leading slice sized from `max_tokens`, cut on a word
-boundary, and grows the budget only if that slice did not reach `max_tokens`.
-The result is identical to tokenizing the whole document, because WordPiece
-segmentation is word-local, but the bytes the tokenizer reads stop growing with
-input size: with `max_tokens: 512` the window is a few kilobytes whether the
-document is 10 KB or 3 MB.
+When truncation is active the runtime starts with a leading slice sized from
+`max_tokens`, cut on a word boundary, and grows the budget until it has reached
+`max_tokens` **usable** ids. For ordinary in-vocabulary text this keeps the
+tokenized prefix to a few kilobytes even when the document is megabytes long.
+For OOV-heavy input it may scan farther — or the whole string — because `[UNK]`
+no longer consumes the usable-token budget. That is required for the corrected
+UNK-before-truncate contract.
 
 That bounds the tokenizer, not the whole call. `embed` also has to establish
 that the Ruby `String` is valid UTF-8, and when Ruby has not computed the
@@ -272,22 +288,34 @@ subword splitting instead of hitting the vocabulary directly.
 
 ## Correctness contract
 
-The reference implementation is `model2vec.StaticModel`. The converter records
-its decisions in the `.semb` file: truncation at 512 tokens by default, applied
-after tokenization and before pooling; `[UNK]` tokens dropped; a zero vector for
-input with no usable tokens; L2 normalization when the source model requires it.
+0.1.5 separates contracts that 0.1.4 accidentally mixed together:
+
+- `tokenize` matches the pinned Hugging Face `tokenizers 0.23.1` raw
+  `BertNormalizer + BertPreTokenizer + WordPiece` ids, including the supported
+  `normalized: false` standard added tokens and `[UNK]`.
+- Embedding drops `[UNK]` when the model records `UNK_DROP`, then applies
+  `max_tokens` to the remaining usable ids, mean-pools, and performs source-model
+  L2 normalization.
+- The gem intentionally does **not** reproduce Model2Vec 0.9.0's character
+  pre-cut (`max_length * median_token_length`). Here `max_tokens` means actual
+  usable tokenizer ids. The oracle reports rows affected by this difference
+  separately instead of calling them parity.
 
-Token ids must match the reference exactly. Vectors are compared with tolerance,
-because floating-point addition order is not bit-stable across implementations:
+Where the execution contracts are the same, vectors are compared to
+`model2vec.StaticModel` with:
 
 ```text
 cosine >= 1 - 1e-6
 max_abs_diff < 1e-5
 ```
 
-`docs/MODEL_AUDIT.md` records the parity result, digests and edge-case
-behaviour for every trusted conversion, and the command to reproduce it. A new
-conversion is not trusted until it has its own record.
+The committed boundary generator currently produces 434 systematic rows. CI
+generates their oracle from pinned Python packages both on the deterministic
+fixture and on the pinned `potion-retrieval-32M` snapshot. The Ruby `Reference`
+fuzz is kept as a separate C-vs-Ruby implementation check and is not treated as
+proof of upstream compatibility. `docs/MODEL_AUDIT.md` records per-model audits;
+a corpus pass means exactly that corpus passed, not an exhaustive proof of all
+Unicode/tokenizer behavior.
 
 ## Development
 
@@ -323,7 +351,7 @@ cc -O2 -std=c99 -Wall -Wextra -Iext/static_embeddings \
   ext/static_embeddings/se_unicode.c \
   ext/static_embeddings/se_tokenizer.c \
   ext/static_embeddings/se_embed.c \
-  -lm -o tmp/memory_smoke
+  -lm -pthread -o tmp/memory_smoke
 ./tmp/memory_smoke tmp/test-tiny.semb
 ```
 

data/Rakefile

@@ -44,7 +44,7 @@ task parity: :compile do
       Generate it first (needs Python and the model2vec package):
 
         python3 -m venv .venv-model2vec && . .venv-model2vec/bin/activate
-        pip install -U model2vec numpy
+        pip install "model2vec==0.9.0" "tokenizers==0.23.1" numpy
         python tools/model2vec_oracle.py /path/to/source-model --out #{oracle}
     MSG
   end

data/docs/ARCHITECTURE.md

@@ -33,7 +33,7 @@ in `se_internal.h`: the header fields are decoded little-endian explicitly, but
 the mmapped structures and the float matrix are read in native order, so a
 big-endian host is refused at load rather than silently misread.
 
-## `.semb` v2
+## `.semb` v3
 
 Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 
@@ -69,6 +69,7 @@ Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 | 208–239 | 32 | two (offset, size) u64 pairs: `root_trie`, `continuation_trie` |
 | 240 | 32 | sha256 of the file with these 32 bytes zeroed |
 | 304 | 4 | max_probe observed in the vocabulary hash |
+| 308 | 4 | supported added-token bit mask |
 
 Sections, in fixed order: `vocab_strings`, `vocab_hash`, `embeddings`,
 `norm_tables`, `provenance`, `root_trie`, `continuation_trie`.
@@ -79,11 +80,12 @@ already has a slot for I8; there is no dead I8 path in C.
 ### Why the semantics live in the header
 
 `max_tokens_default`, `unk_policy`, `empty_policy` and the normalizer flags
-are properties of the *model*, not options the caller guesses. The reference
-implementation truncates at 512 tokens; a model card's `model_max_length` of
-1,000,000 is a different number entirely, and confusing them silently changes
-every vector for long chunks. Anything that can silently change output is a
-recorded field.
+are properties of the *model*, not options the caller guesses. The embedding contract caps the first 512 usable token ids by default;
+a model card's `model_max_length` of 1,000,000 is a different number entirely, and
+confusing them silently changes every vector for long chunks. Model2Vec 0.9.0
+also applies a character pre-cut before tokenization; this runtime intentionally
+does not copy that heuristic. Anything that can silently change output is a
+recorded field or an explicitly documented compatibility deviation.
 
 ## Vocabulary lookup and WordPiece trie
 
@@ -93,7 +95,7 @@ load factor ≤ 0.70, linear probing, FNV-1a with a fixed seed. The hash table i
 kept for validation, direct lookup and debugging.
 
 The hot WordPiece path no longer performs repeated hash lookups for every
-`end--` candidate. Format v2 also stores two converter-built, mmap-readable
+`end--` candidate. The format also stores two converter-built, mmap-readable
 tries inspired by double-array trie libraries such as libdatrie:
 
 - `root_trie` for tokens that can start a word;
@@ -135,8 +137,8 @@ Rejected alternatives:
 
 ## Unicode without a Unicode library
 
-`BertNormalizer` needs NFD, simple lowercase, and the Mn / P / C / Zs
-categories. Rather than vendoring utf8proc, the converter generates exactly
+`BertNormalizer` needs NFD, simple lowercase, and selected Unicode
+categories (Mn, punctuation, control/format/private-use, and whitespace). Rather than vendoring utf8proc, the converter generates exactly
 those tables from the Ruby VM's own Unicode data and writes them into the
 model file; the runtime binary-searches them.
 
@@ -146,15 +148,20 @@ maps `ß` to `ss`; `str::to_lowercase`, which HF uses, does not. A gem that
 picked the wrong one would produce plausible, slightly wrong vectors forever.
 
 The Ruby build that generated the tables is stamped into provenance, so a
-model file and its normalizer can never drift apart unnoticed.
+model file and its embedded normalizer tables can never drift apart unnoticed.
+That stamp is not, by itself, proof of Hugging Face parity: 0.1.5 therefore
+checks a systematic boundary corpus against pinned `tokenizers 0.23.1` in CI.
+One subtle example is `Cn`: `bert.rs` says `is_other()` includes unassigned
+codepoints, while the `unicode_categories 0.1.1` implementation actually tests
+Cc/Cf/Co. The runtime follows the executable upstream behavior, not the comment.
 
 ## The embedding kernel
 
 ```
-tokenize -> ids
-truncate to max_tokens          (before pooling — the reference contract)
+tokenize -> raw ids (including [UNK])
 drop [UNK] if unk_policy = DROP
-acc[j] += row[j] for each id, in token order
+truncate usable ids to max_tokens
+acc[j] += row[j] for each usable id, in token order
 divide by the number of pooled rows
 L2 normalize if the model says so
 ```
@@ -174,10 +181,13 @@ storage/transport choice, not a different model. `embed_array` and
 
 The f32 compute path uses small explicit SIMD kernels where they keep the format
 simple: row accumulation and scaling use NEON/SSE when the compiler target
-exposes them, with scalar fallback everywhere else. `dot_top_k`/`cosine_top_k`
+exposes them, L2 sum-of-squares uses SIMD double (NEON on AArch64, SSE2 on
+x86) so the inverse stays a `1/sqrt` of a double accumulation, and there is a
+scalar fallback everywhere else. `dot_top_k`/`cosine_top_k`
 share the SIMD dot kernel for `format: :f32`; `cosine_top_k` additionally
 accumulates each row's sum of squares so it can divide by the true norms.
-`format: :f16` top-k decodes half components on the fly. What is still
+`format: :f16` top-k decodes half components on the fly (16-wide on AArch64
+NEON-FP16, 16-wide on x86 F16C). What is still
 deliberately absent is a dependency on BLAS: pooling is gathering random
 embedding rows, not a dense matrix multiply.
 
@@ -206,10 +216,11 @@ therefore serialised the process on work that the tokenizer would discard after
 
 `embed` and `embed_batch` instead copy a leading slice. The budget starts at
 `max_tokens * 16` bytes clamped to `[4096, 65536]`, and the slice is cut at a
-position where the tokenizer would have started a fresh word. A result is
-accepted only when that slice actually reached `max_tokens`; otherwise the
-budget doubles and the text is tokenized again. Growth is geometric, so the
-pathological case costs about twice the single-pass work.
+position where the tokenizer would have started a fresh word. A result is accepted only when that slice actually reached `max_tokens`
+usable ids; `[UNK]` does not consume the embedding budget. Otherwise the budget
+doubles and the text is tokenized again. Growth is geometric. OOV-heavy input
+may therefore reach the full document: deciding that fewer than `max_tokens`
+usable ids exist requires reading it.
 
 Three properties make this exact rather than approximate:
 
@@ -238,10 +249,11 @@ the predicate is Unicode-aware. Text with genuinely no legal cut inside the scan
 window - one enormous word, a run of combining marks - still falls back to a
 full copy, because there is nothing else that would be correct.
 
-`test/prefix_chunking_test.rb` sweeps every printable ASCII separator, checks
-CJK, NBSP and Unicode-punctuation documents against
-`embed_token_ids(tokenize(text))`, and asserts that quadrupling the input does
-not triple the cost.
+`test/prefix_chunking_test.rb` sweeps every printable ASCII separator and
+checks CJK, NBSP and Unicode-punctuation documents against unbounded raw
+tokenization followed by `embed_token_ids`. It asserts bounded prefix cost for
+in-vocabulary text; deliberately OOV-heavy input is checked for correctness, not
+flat cost.
 
 With `max_tokens: false` there is nothing to truncate and the whole input is
 copied.
@@ -293,8 +305,16 @@ cadence in bytes so a long ASCII run is no less interruptible. The pooling loop
 checks every 256 rows, and the top-k scan every 1024 rows. `unblock_cancel` sets that flag when Ruby
 interrupts a GVL-free region.
 
-No global mutable state exists in C. The model is immutable after load and
-scratch buffers are per call.
+No global mutable state exists in C. The model is immutable after load.
+Scratch buffers are reused per OS thread via `pthread_key` / `FlsAlloc`, and
+the destructor frees them when the thread exits — including the short-lived
+threads the fiber scheduler path creates per large call. On release the
+codepoint/byte buffers are trimmed back to their reserve sizes. The token-id
+buffer keeps its observed high-water mark up to 8192 ids so repeated OOV-heavy
+usable-token truncation does not grow and shrink it on every call; larger
+capacities are trimmed back to that bound so one unlimited-`max_tokens`
+document cannot pin an arbitrary per-thread working set. A nested call on the
+same thread allocates a one-off heap scratch that is freed on release.
 
 ## Where this fits in a RAG pipeline
 
@@ -313,12 +333,11 @@ chunks -> .semb vectors --/
 
 ## Open questions before v1.0
 
-1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The 32M family
-   has a larger vocabulary than the bge-base tokenizer it was distilled from,
-   which means tokens were added somewhere. If they live in `added_tokens`
-   rather than `model.vocab`, HF matches them with a separate trie pass over
-   raw text that this runtime does not implement. The converter refuses such
-   models today; whether it has to is an audit question.
+1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The runtime now
+   implements the exact `normalized: false` extraction semantics needed by the
+   five standard BERT added tokens, but still rejects arbitrary AddedVocabulary,
+   BPE and Unigram profiles. Each source model still needs an audit before it is
+   treated as compatible.
 2. **Russian.** Distilling a multilingual teacher into a WordPiece vocabulary
    we control keeps the pure-C path, but skips the Tokenlearn pre-training
    that gives the published potion models their quality. That gap has to be

data/docs/LIMITATIONS.md

@@ -13,9 +13,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Only `.semb` files from this repository's converter.** Loading a
   HuggingFace directory at runtime is not supported and will not be. The
   converter is the only thing that reads third-party files, and it runs offline.
-- **Only the `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE, Unigram
-  and byte-level tokenizers are rejected at conversion time rather than
-  approximated.
+- **Only the pinned `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE,
+  Unigram and byte-level tokenizers are rejected. AddedVocabulary support is
+  deliberately limited to the five standard BERT special tokens with
+  `special: true`, `normalized: false` and no strip/single-word behavior.
 - **No model training, distillation or fine-tuning.** Produce the model with
   upstream `model2vec`, then convert it.
 - **No transformer inference.** Static embeddings have no attention and no
@@ -38,6 +39,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Big-endian hosts are refused at load.** The header is decoded
   little-endian explicitly, but the mmapped structures and the float matrix are
   read in native order.
+- **Do not truncate/overwrite a mapped `.semb` in place.** On POSIX, shrinking
+  the inode underneath an existing mmap can raise `SIGBUS` in a worker. Deploy a
+  new file and atomically rename it over the path (`write .tmp` → `fsync` →
+  `rename`); existing workers keep their old inode mapping safely.
 
 ## Search
 
@@ -57,8 +62,10 @@ the entry says whether that is a decision or just unfinished work.
 ## Storage formats
 
 - **`f32` and `f16` only.** No int8, no binary quantisation.
-- **`f16` is a storage trade-off, not a speed feature.** See
-  `docs/PERFORMANCE.md`; it wins on some CPUs and loses on others.
+- **`f16` is a storage encoding, not a different model.** Top-k over an f16
+  matrix can be faster or slower than f32 depending on the decode kernel; see
+  `docs/PERFORMANCE.md`. `embed_batch(format: :f16)` still encodes the blob
+  with the scalar converter.
 - **`f16` rounding is half-up, not ties-to-even.** Blobs written by this gem can
   differ from NumPy or PyTorch by one ULP on exact halfway values.
 
@@ -74,9 +81,10 @@ the entry says whether that is a decision or just unfinished work.
 - **`load` does not check the SHA-256.** Structural validation always runs, but
   a bit flip inside the matrix is only caught by `verify: true` or
   `StaticEmbeddings.verify`.
-- **Parity is per model and per runtime version.** A converted model is not
-  trusted until it has a record in `docs/MODEL_AUDIT.md`, and that record
-  expires when the tokenizer changes.
+- **Parity is per model, runtime, pinned upstream versions and corpus.** A
+  recorded pass means those rows passed; it is not an exhaustive proof of
+  tokenizer equivalence. The independent upstream CI and the Ruby implementation
+  fuzz intentionally test different things.
 
 ## Known open questions