static_embeddings 0.1.4 → 1.5.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f72b45c2d6fa5310e3b6d5a9b0a86ab0430fc4af71f6811f9aa02655de6272a8
-  data.tar.gz: 51f0bb96a3b0adc5f2799e1df425cf1747ca521d92ae90afba59e0014f1d4106
+  metadata.gz: 550531f7573a8d05e8d5a3a1ae0fa301c01d77b238573ae11834797e8ac44a0f
+  data.tar.gz: 57f487c729296e3a858604dc64aa146aa6dd6e372a45961d90a61564e3c1b117
 SHA512:
-  metadata.gz: 53630b742cd2723656b5d826a82f57042dcaab1e3259758aa6a71eac20ba387a23b3fb3905f0f36f3ce641b122229f366c9af1f029205b44cd82110f538a49cb
-  data.tar.gz: 199cd93e9898b7457d5e19b9cf1857b4d78ffc46777ca26778b094fa70ee02559dc90733e4e815eba74297ef9c066940792cef09e022801ef48c704e868fa8db
+  metadata.gz: 52162c96d458dffaaec8498cd712678b4727b29ddaa66eadf7834893113a6bd05f636cc47a2f84d969f76eb4651fe57664212fdb4ac2925449827b9d3a69a9bf
+  data.tar.gz: 115a0b0f3ff9d82cea099fc0ee24385034d8bcc66ca4ff9337f2213608ee4d611215b6d3400fe39aecf562cf201f5f5044eab7f9863fbdc813362c0af0eb7739

data/CHANGELOG.md

@@ -1,5 +1,123 @@
 # Changelog
 
+## 1.5.6
+
+Multi-source WordPiece release. The native `.semb` v3 runtime is unchanged; new
+source families are normalized offline into the same audited
+`BERT_WORDPIECE_V1 -> lookup -> mean -> NONE/L2` contract. Model identity is
+chosen at `load_model`, not per `embed` call.
+
+### Added
+
+- **Model2Vec and Sentence Transformers importers.** A Model2Vec directory and a
+  Sentence Transformers `StaticEmbedding` directory (`modules.json` + exactly
+  one module path) compile to the same canonical data. The ST importer is
+  fail-closed and confines the module path to the source root with `realpath`.
+- **Source-faithful policies.** Model2Vec uses `UNK_DROP`, normalization from its
+  config and a 512-token default. Sentence Transformers `StaticEmbedding` uses
+  `UNK_INCLUDE`, `NORMALIZATION_NONE`, unlimited tokens and
+  `add_special_tokens=false`.
+- **`--dimensions N`.** Matryoshka prefix slicing happens while streaming the
+  embedding matrix; the runtime still loads an ordinary fixed-dimension `.semb`.
+- **`--max-tokens unlimited`.** Convert-time `0` remains an alias; runtime
+  `max_tokens: 0` is still rejected and `false` means unlimited.
+- **Russian retrieval fixture.** `tools/eval_retrieval.rb` and the checked-in
+  Russian FAQ corpus provide a small domain sanity check for the multilingual
+  WordPiece model; it is not presented as a general benchmark.
+
+### Changed
+
+- **Offline Ruby conversion was simplified.** Stateful `Converter` and the
+  behavior-heavy `CanonicalModel` were replaced by import functions, immutable
+  canonical data, pure provenance/meta transforms and a streaming format writer.
+  Writer/hash/trie/verifier code is split from runtime format constants, so plain
+  `require "static_embeddings"` does not load conversion code.
+- The Ruby `Reference` twin follows canonical UNK/normalization/max-token
+  policies instead of re-deriving Model2Vec defaults.
+- Model2Vec `normalize` now defaults to `false` when the key is absent, matching
+  the pinned upstream behavior. Unknown source families are rejected rather than
+  guessed from `tokenizer.json + model.safetensors`.
+- Header integer writes validate their unsigned range instead of silently
+  wrapping oversized Ruby integers.
+- `Model#provenance` is parsed once and cached.
+
+Local WordPiece conversions requiring **no C change**: `potion-base-8M`,
+`potion-science-32M`, `static-retrieval-mrl-en-v1` at 1024/512 and
+`static-similarity-mrl-multilingual-v1` at 512/256. The retrieval model has a
+recorded `SentenceTransformer.encode` oracle (436/436). Records are in
+`docs/MODEL_AUDIT.md`.
+
+SentencePiece/Unigram is intentionally **not** part of 1.5.6; it remains a
+future runtime capability rather than shipping an unaudited v4 path.
+
+## 0.1.5
+
+Correctness/parity release. The native runtime architecture is unchanged, but
+the tokenizer/pooling contract is now pinned explicitly to `model2vec 0.9.0` +
+`tokenizers 0.23.1`, and CI contains an independent upstream boundary corpus
+instead of treating the Ruby `Reference` twin as proof of compatibility.
+
+### Fixed
+
+- **`[UNK]` is filtered before the embedding token cap.** `max_tokens: 512` now
+  means the first 512 usable (non-UNK, for `UNK_DROP`) ids, matching
+  `StaticModel` pooling semantics. Long OOV-heavy text can therefore scan past
+  512 raw ids instead of silently averaging fewer rows. `embed_with_stats` and
+  `embed_token_ids_with_stats` expose `pooled_count` alongside raw
+  `token_count` / `unk_count`.
+- **The supported Hugging Face AddedVocabulary subset is implemented.** The five
+  standard BERT special tokens (`[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]`)
+  are extracted literally before normalization when the source tokenizer marks
+  them `special: true, normalized: false`. Converter and runtime now share that
+  contract instead of merely whitelisting the metadata.
+- **BertNormalizer behavior matches the pinned Rust tokenizer at the known edges.**
+  The CJK Extension E range starts at `U+2B920` as in `tokenizers 0.23.1`, not
+  the Python `transformers` boundary `U+2B820`. The boundary corpus also locks
+  the actual `unicode_categories` 0.1.1 behavior for unassigned (`Cn`)
+  codepoints: despite a misleading comment in upstream `bert.rs`, that crate's
+  `is_other()` implementation checks Cc/Cf/Co, not Cn.
+- **Vocabulary hash lookup is bounded by the validated `max_probe`,** and model
+  loading rejects hash tables above the converter's 0.70 load factor.
+- **`load_model` model ids cannot escape the cache** through absolute paths,
+  `..`, empty path components, backslash traversal or NUL bytes.
+
+### Changed
+
+- **`.semb` format v3.** The header records the supported added-token mask and
+  the usable-id truncation policy. v2 files must be reconverted; this avoids a
+  runtime silently applying 0.1.5 semantics to a file whose tokenizer contract
+  was recorded by an older converter.
+- **Model2Vec's character pre-truncation heuristic is intentionally not copied.**
+  `max_tokens` means actual usable tokenizer ids. The external oracle records
+  rows where `StaticModel`'s `max_length * median_token_length` character cut
+  changes ids and reports them as an explicit semantic deviation rather than a
+  false parity failure/pass.
+- **The converter reads safetensors incrementally and accepts F32, F16 and BF16
+  embedding tensors.** `.semb` rows remain float32. `Format.write` also streams
+  sections and computes the checksum incrementally instead of assembling a
+  second whole-file Ruby string.
+- **Plain `require "static_embeddings"` loads runtime code only.** Converter,
+  safetensors, Unicode-table generation and the Ruby `Reference` are loaded only
+  by offline tooling/tests. The `static_embeddings convert` CLI explicitly loads
+  the converter before reading converter-specific defaults, so lazy loading does
+  not break the documented executable path.
+- **TLS token-id scratch keeps a bounded high-water mark.** It still reserves only
+  512 ids initially, but capacities grown by OOV-heavy usable-token truncation are
+  retained up to 8192 ids between calls. This removes grow/trim realloc churn on
+  realistic truncated inputs without pinning arbitrarily large per-thread buffers.
+
+### Verification
+
+- Added a deterministic **434-row boundary corpus** covering control/unassigned
+  boundaries, every supported CJK range edge ±1, punctuation classes, whitespace, combining
+  marks, AddedVocabulary, 511/512/513 boundaries, OOV ratios and long-prefix
+  cases.
+- CI now installs pinned `model2vec==0.9.0` / `tokenizers==0.23.1`, generates an
+  external oracle, and separately checks raw HF ids, usable ids, embedding
+  parity where the contracts are identical, and intentional character-precut
+  deviations. The existing 50k native-vs-Ruby differential fuzz remains as a
+  separate implementation-consistency layer.
+
 ## 0.1.4 (unreleased)
 
 Hot-path work in the C runtime. Token ids, the pooling contract and f16

data/README.md

@@ -1,11 +1,11 @@
 # static_embeddings
 
-A Ruby runtime for converted Model2Vec / potion static embedding models. A model
+A Ruby runtime for converted Model2Vec and Sentence Transformers static embedding models. A model
 is converted once into a local `.semb` file and loaded through a small C
 extension. No ONNX Runtime, no Rust or Python at runtime, no network access, one
 mmap-able file, binary float32 output.
 
-The current production target is `minishlab/potion-retrieval-32M`.
+The reference production target remains `minishlab/potion-retrieval-32M`; 1.5.6 also supports audited WordPiece `StaticEmbedding` sources.
 
 ```ruby
 require "static_embeddings"
@@ -32,8 +32,11 @@ never arbitrary HuggingFace models. A `.semb` file carries validated metadata, a
 BERT WordPiece tokenizer profile, an mmap-ready vocabulary lookup, float32
 embedding rows, and provenance plus a checksum.
 
-The only supported tokenizer profile is `BERT_WORDPIECE_V1`. The converter
-rejects unsupported tokenizer features rather than approximating them.
+The only supported tokenizer profile is `BERT_WORDPIECE_V1`, pinned for
+compatibility to `tokenizers 0.23.1`. The converter rejects unsupported
+tokenizer features rather than approximating them. `.semb` format v3 also
+records which of the five standard BERT added tokens are active. Files produced
+by 0.1.4 and earlier use format v2 and must be reconverted.
 
 ## Converting a model
 
@@ -46,6 +49,21 @@ bundle exec ruby -Ilib exe/static_embeddings convert ./potion-retrieval-32M \
   --id potion-retrieval-32m
 ```
 
+Sentence Transformers static models (`modules.json` + one `StaticEmbedding`
+module) use the same command. Matryoshka prefix-slice is a convert-time flag,
+not a runtime `dim:`:
+
+```bash
+bundle exec ruby -Ilib exe/static_embeddings convert ./static-retrieval-mrl-en-v1 \
+  --id static-retrieval-mrl-en-v1-512 \
+  --dimensions 512 \
+  --trained-mrl-dims 1024,512,256,128,64,32
+```
+
+`--max-tokens unlimited` bakes no token cap (Sentence Transformers default).
+`--max-tokens 0` is a convert-time alias; `embed(max_tokens: 0)` is still
+rejected. Query and documents must use the same loaded `.semb`.
+
 The result lands in `~/.cache/static_embeddings/models/potion-retrieval-32m.semb`.
 Inspect or verify it with the `inspect` and `verify` subcommands, then load it:
 
@@ -57,6 +75,9 @@ model = StaticEmbeddings.load(ENV.fetch("EMBEDDING_MODEL"))   # explicit path
 Convert during image build or deploy preparation; production should only ever
 see `.semb` files.
 
+The offline converter accepts F32, F16 and BF16 safetensors embedding matrices;
+all three are converted to float32 rows in `.semb`.
+
 ### Verify once, not on every boot
 
 `load` does not check the SHA-256 by default: hashing a 135 MB file at boot
@@ -98,6 +119,7 @@ stats = model.embed_with_stats("postgres pipeline mode in Ruby")
 stats[:vector]
 stats[:token_count]
 stats[:unk_count]
+stats[:pooled_count]  # rows actually eligible for pooling
 stats[:truncated]
 
 ids         = model.tokenize("postgres pipeline mode in Ruby")
@@ -116,11 +138,19 @@ Output is a binary `String` of little-endian float32 values in row-major order.
 `embed_array` and `embed_batch_arrays` decode that into Ruby `Float` objects and
 exist for debugging and application code, not for the hot path.
 
-`embed_token_ids` pools ids you supply, applying the same `max_tokens`
-truncation as `embed`, so `embed_token_ids(model.tokenize(text))` equals
-`embed(text)`. Pass `max_tokens: false` to pool every id. It is for reusing a
-cached tokenization and for benchmarking pooling in isolation; it is not a
-faster path for ordinary text.
+`tokenize` exposes raw WordPiece ids, including `[UNK]`, and its own
+`max_tokens` cap is therefore a raw-id cap. Embedding has a different, upstream
+pooling rule: for `UNK_DROP`, `[UNK]` is removed first and `max_tokens` is then
+applied to usable ids. To cache tokenization without changing the embedding,
+cache it unbounded:
+
+```ruby
+ids = model.tokenize(text, max_tokens: false)
+model.embed_token_ids(ids) == model.embed(text)
+```
+
+`embed_token_ids` applies that same usable-id cap. This distinction matters only
+when `[UNK]` appears around the truncation boundary.
 
 ### `format: :f16`
 
@@ -175,13 +205,13 @@ model.dot_top_k(query, MATRIX, 10)          # lock-free, concurrent, GVL release
 
 ## Large inputs
 
-When truncation is active the runtime does not copy the whole string into C
-memory. It copies a leading slice sized from `max_tokens`, cut on a word
-boundary, and grows the budget only if that slice did not reach `max_tokens`.
-The result is identical to tokenizing the whole document, because WordPiece
-segmentation is word-local, but the bytes the tokenizer reads stop growing with
-input size: with `max_tokens: 512` the window is a few kilobytes whether the
-document is 10 KB or 3 MB.
+When truncation is active the runtime starts with a leading slice sized from
+`max_tokens`, cut on a word boundary, and grows the budget until it has reached
+`max_tokens` **usable** ids. For ordinary in-vocabulary text this keeps the
+tokenized prefix to a few kilobytes even when the document is megabytes long.
+For OOV-heavy input it may scan farther — or the whole string — because `[UNK]`
+no longer consumes the usable-token budget. That is required for the corrected
+UNK-before-truncate contract.
 
 That bounds the tokenizer, not the whole call. `embed` also has to establish
 that the Ruby `String` is valid UTF-8, and when Ruby has not computed the
@@ -273,22 +303,34 @@ subword splitting instead of hitting the vocabulary directly.
 
 ## Correctness contract
 
-The reference implementation is `model2vec.StaticModel`. The converter records
-its decisions in the `.semb` file: truncation at 512 tokens by default, applied
-after tokenization and before pooling; `[UNK]` tokens dropped; a zero vector for
-input with no usable tokens; L2 normalization when the source model requires it.
+0.1.5 separates contracts that 0.1.4 accidentally mixed together:
+
+- `tokenize` matches the pinned Hugging Face `tokenizers 0.23.1` raw
+  `BertNormalizer + BertPreTokenizer + WordPiece` ids, including the supported
+  `normalized: false` standard added tokens and `[UNK]`.
+- Embedding drops `[UNK]` when the model records `UNK_DROP`, then applies
+  `max_tokens` to the remaining usable ids, mean-pools, and performs source-model
+  L2 normalization.
+- The gem intentionally does **not** reproduce Model2Vec 0.9.0's character
+  pre-cut (`max_length * median_token_length`). Here `max_tokens` means actual
+  usable tokenizer ids. The oracle reports rows affected by this difference
+  separately instead of calling them parity.
 
-Token ids must match the reference exactly. Vectors are compared with tolerance,
-because floating-point addition order is not bit-stable across implementations:
+Where the execution contracts are the same, vectors are compared to
+`model2vec.StaticModel` with:
 
 ```text
 cosine >= 1 - 1e-6
 max_abs_diff < 1e-5
 ```
 
-`docs/MODEL_AUDIT.md` records the parity result, digests and edge-case
-behaviour for every trusted conversion, and the command to reproduce it. A new
-conversion is not trusted until it has its own record.
+The committed boundary generator currently produces 434 systematic rows. CI
+generates their oracle from pinned Python packages both on the deterministic
+fixture and on the pinned `potion-retrieval-32M` snapshot. The Ruby `Reference`
+fuzz is kept as a separate C-vs-Ruby implementation check and is not treated as
+proof of upstream compatibility. `docs/MODEL_AUDIT.md` records per-model audits;
+a corpus pass means exactly that corpus passed, not an exhaustive proof of all
+Unicode/tokenizer behavior.
 
 ## Development
 

data/Rakefile

@@ -44,7 +44,7 @@ task parity: :compile do
       Generate it first (needs Python and the model2vec package):
 
         python3 -m venv .venv-model2vec && . .venv-model2vec/bin/activate
-        pip install -U model2vec numpy
+        pip install "model2vec==0.9.0" "tokenizers==0.23.1" numpy
         python tools/model2vec_oracle.py /path/to/source-model --out #{oracle}
     MSG
   end

data/docs/ARCHITECTURE.md

@@ -8,10 +8,10 @@ HuggingFace / Model2Vec files          offline, once, on your machine
   model.safetensors
         |
         v
-  Converter (pure Ruby, strict)   <--- all parsing, all validation, all
-        |                              Unicode table generation happens here
+  import -> canonical data -> writer   <--- pure Ruby, offline, strict
+        |                                    all source parsing/validation here
         v
-  model.semb                       <--- flat, versioned, mmap-able
+  model.semb                            <--- flat, versioned, mmap-able
         |
         v
   C runtime                        <--- mmap + bounds checks + tokenize +
@@ -24,6 +24,15 @@ Everything expensive, fragile or security-sensitive about reading third-party
 model files happens once, offline, in a language where it is easy to get
 right. What remains in C is a bounds-checked mmap and three loops.
 
+Only the offline import layer knows HuggingFace layouts. Model2Vec and Sentence
+Transformers `StaticEmbedding` sources are reduced to immutable canonical data:
+tokens, streamed matrix, tokenizer metadata, runtime policies and provenance.
+Pure transforms then build the v3 header/provenance and the existing WordPiece
+writer emits the artifact. A new WordPiece + mean-pooling source therefore does
+not require a C change. New tokenizer families remain explicit future
+capabilities and require a separately audited format/runtime change rather than
+source-name special cases.
+
 The C side is `se_format.c` (mmap and validation), `se_tokenizer.c`,
 `se_embed.c`, `se_unicode.c`, `se_f16.c` (half-precision codec and kernels),
 `se_topk.c`, `se_alloc_stats.c` (optional allocation counters), and
@@ -33,7 +42,7 @@ in `se_internal.h`: the header fields are decoded little-endian explicitly, but
 the mmapped structures and the float matrix are read in native order, so a
 big-endian host is refused at load rather than silently misread.
 
-## `.semb` v2
+## `.semb` v3
 
 Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 
@@ -69,6 +78,7 @@ Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 | 208–239 | 32 | two (offset, size) u64 pairs: `root_trie`, `continuation_trie` |
 | 240 | 32 | sha256 of the file with these 32 bytes zeroed |
 | 304 | 4 | max_probe observed in the vocabulary hash |
+| 308 | 4 | supported added-token bit mask |
 
 Sections, in fixed order: `vocab_strings`, `vocab_hash`, `embeddings`,
 `norm_tables`, `provenance`, `root_trie`, `continuation_trie`.
@@ -79,11 +89,12 @@ already has a slot for I8; there is no dead I8 path in C.
 ### Why the semantics live in the header
 
 `max_tokens_default`, `unk_policy`, `empty_policy` and the normalizer flags
-are properties of the *model*, not options the caller guesses. The reference
-implementation truncates at 512 tokens; a model card's `model_max_length` of
-1,000,000 is a different number entirely, and confusing them silently changes
-every vector for long chunks. Anything that can silently change output is a
-recorded field.
+are properties of the *model*, not options the caller guesses. The embedding contract caps the first 512 usable token ids by default;
+a model card's `model_max_length` of 1,000,000 is a different number entirely, and
+confusing them silently changes every vector for long chunks. Model2Vec 0.9.0
+also applies a character pre-cut before tokenization; this runtime intentionally
+does not copy that heuristic. Anything that can silently change output is a
+recorded field or an explicitly documented compatibility deviation.
 
 ## Vocabulary lookup and WordPiece trie
 
@@ -93,7 +104,7 @@ load factor ≤ 0.70, linear probing, FNV-1a with a fixed seed. The hash table i
 kept for validation, direct lookup and debugging.
 
 The hot WordPiece path no longer performs repeated hash lookups for every
-`end--` candidate. Format v2 also stores two converter-built, mmap-readable
+`end--` candidate. The format also stores two converter-built, mmap-readable
 tries inspired by double-array trie libraries such as libdatrie:
 
 - `root_trie` for tokens that can start a word;
@@ -135,8 +146,8 @@ Rejected alternatives:
 
 ## Unicode without a Unicode library
 
-`BertNormalizer` needs NFD, simple lowercase, and the Mn / P / C / Zs
-categories. Rather than vendoring utf8proc, the converter generates exactly
+`BertNormalizer` needs NFD, simple lowercase, and selected Unicode
+categories (Mn, punctuation, control/format/private-use, and whitespace). Rather than vendoring utf8proc, the converter generates exactly
 those tables from the Ruby VM's own Unicode data and writes them into the
 model file; the runtime binary-searches them.
 
@@ -146,15 +157,20 @@ maps `ß` to `ss`; `str::to_lowercase`, which HF uses, does not. A gem that
 picked the wrong one would produce plausible, slightly wrong vectors forever.
 
 The Ruby build that generated the tables is stamped into provenance, so a
-model file and its normalizer can never drift apart unnoticed.
+model file and its embedded normalizer tables can never drift apart unnoticed.
+That stamp is not, by itself, proof of Hugging Face parity: 0.1.5 therefore
+checks a systematic boundary corpus against pinned `tokenizers 0.23.1` in CI.
+One subtle example is `Cn`: `bert.rs` says `is_other()` includes unassigned
+codepoints, while the `unicode_categories 0.1.1` implementation actually tests
+Cc/Cf/Co. The runtime follows the executable upstream behavior, not the comment.
 
 ## The embedding kernel
 
 ```
-tokenize -> ids
-truncate to max_tokens          (before pooling — the reference contract)
+tokenize -> raw ids (including [UNK])
 drop [UNK] if unk_policy = DROP
-acc[j] += row[j] for each id, in token order
+truncate usable ids to max_tokens
+acc[j] += row[j] for each usable id, in token order
 divide by the number of pooled rows
 L2 normalize if the model says so
 ```
@@ -209,10 +225,11 @@ therefore serialised the process on work that the tokenizer would discard after
 
 `embed` and `embed_batch` instead copy a leading slice. The budget starts at
 `max_tokens * 16` bytes clamped to `[4096, 65536]`, and the slice is cut at a
-position where the tokenizer would have started a fresh word. A result is
-accepted only when that slice actually reached `max_tokens`; otherwise the
-budget doubles and the text is tokenized again. Growth is geometric, so the
-pathological case costs about twice the single-pass work.
+position where the tokenizer would have started a fresh word. A result is accepted only when that slice actually reached `max_tokens`
+usable ids; `[UNK]` does not consume the embedding budget. Otherwise the budget
+doubles and the text is tokenized again. Growth is geometric. OOV-heavy input
+may therefore reach the full document: deciding that fewer than `max_tokens`
+usable ids exist requires reading it.
 
 Three properties make this exact rather than approximate:
 
@@ -241,10 +258,11 @@ the predicate is Unicode-aware. Text with genuinely no legal cut inside the scan
 window - one enormous word, a run of combining marks - still falls back to a
 full copy, because there is nothing else that would be correct.
 
-`test/prefix_chunking_test.rb` sweeps every printable ASCII separator, checks
-CJK, NBSP and Unicode-punctuation documents against
-`embed_token_ids(tokenize(text))`, and asserts that quadrupling the input does
-not triple the cost.
+`test/prefix_chunking_test.rb` sweeps every printable ASCII separator and
+checks CJK, NBSP and Unicode-punctuation documents against unbounded raw
+tokenization followed by `embed_token_ids`. It asserts bounded prefix cost for
+in-vocabulary text; deliberately OOV-heavy input is checked for correctness, not
+flat cost.
 
 With `max_tokens: false` there is nothing to truncate and the whole input is
 copied.
@@ -299,10 +317,13 @@ interrupts a GVL-free region.
 No global mutable state exists in C. The model is immutable after load.
 Scratch buffers are reused per OS thread via `pthread_key` / `FlsAlloc`, and
 the destructor frees them when the thread exits — including the short-lived
-threads the fiber scheduler path creates per large call. On release the slot
-is trimmed back to the reserve sizes, so a single unlimited-`max_tokens`
-document does not pin its working set until the thread dies. A nested call on
-the same thread allocates a one-off heap scratch that is freed on release.
+threads the fiber scheduler path creates per large call. On release the
+codepoint/byte buffers are trimmed back to their reserve sizes. The token-id
+buffer keeps its observed high-water mark up to 8192 ids so repeated OOV-heavy
+usable-token truncation does not grow and shrink it on every call; larger
+capacities are trimmed back to that bound so one unlimited-`max_tokens`
+document cannot pin an arbitrary per-thread working set. A nested call on the
+same thread allocates a one-off heap scratch that is freed on release.
 
 ## Where this fits in a RAG pipeline
 
@@ -321,12 +342,11 @@ chunks -> .semb vectors --/
 
 ## Open questions before v1.0
 
-1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The 32M family
-   has a larger vocabulary than the bge-base tokenizer it was distilled from,
-   which means tokens were added somewhere. If they live in `added_tokens`
-   rather than `model.vocab`, HF matches them with a separate trie pass over
-   raw text that this runtime does not implement. The converter refuses such
-   models today; whether it has to is an audit question.
+1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The runtime now
+   implements the exact `normalized: false` extraction semantics needed by the
+   five standard BERT added tokens, but still rejects arbitrary AddedVocabulary,
+   BPE and Unigram profiles. Type A (WordPiece + mean) models convert without a
+   C change; each source still needs an audit before it is treated as compatible.
 2. **Russian.** Distilling a multilingual teacher into a WordPiece vocabulary
    we control keeps the pure-C path, but skips the Tokenlearn pre-training
    that gives the published potion models their quality. That gap has to be

data/docs/LIMITATIONS.md

@@ -13,9 +13,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Only `.semb` files from this repository's converter.** Loading a
   HuggingFace directory at runtime is not supported and will not be. The
   converter is the only thing that reads third-party files, and it runs offline.
-- **Only the `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE, Unigram
-  and byte-level tokenizers are rejected at conversion time rather than
-  approximated.
+- **Only the pinned `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE,
+  Unigram and byte-level tokenizers are rejected. AddedVocabulary support is
+  deliberately limited to the five standard BERT special tokens with
+  `special: true`, `normalized: false` and no strip/single-word behavior.
 - **No model training, distillation or fine-tuning.** Produce the model with
   upstream `model2vec`, then convert it.
 - **No transformer inference.** Static embeddings have no attention and no
@@ -38,6 +39,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Big-endian hosts are refused at load.** The header is decoded
   little-endian explicitly, but the mmapped structures and the float matrix are
   read in native order.
+- **Do not truncate/overwrite a mapped `.semb` in place.** On POSIX, shrinking
+  the inode underneath an existing mmap can raise `SIGBUS` in a worker. Deploy a
+  new file and atomically rename it over the path (`write .tmp` → `fsync` →
+  `rename`); existing workers keep their old inode mapping safely.
 
 ## Search
 
@@ -76,9 +81,10 @@ the entry says whether that is a decision or just unfinished work.
 - **`load` does not check the SHA-256.** Structural validation always runs, but
   a bit flip inside the matrix is only caught by `verify: true` or
   `StaticEmbeddings.verify`.
-- **Parity is per model and per runtime version.** A converted model is not
-  trusted until it has a record in `docs/MODEL_AUDIT.md`, and that record
-  expires when the tokenizer changes.
+- **Parity is per model, runtime, pinned upstream versions and corpus.** A
+  recorded pass means those rows passed; it is not an exhaustive proof of
+  tokenizer equivalence. The independent upstream CI and the Ruby implementation
+  fuzz intentionally test different things.
 
 ## Known open questions