static_embeddings 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f72b45c2d6fa5310e3b6d5a9b0a86ab0430fc4af71f6811f9aa02655de6272a8
-  data.tar.gz: 51f0bb96a3b0adc5f2799e1df425cf1747ca521d92ae90afba59e0014f1d4106
+  metadata.gz: fb1d3530fba4fad467251429797fdcaae527b5c6c18413e8847002e0c3f3b5e4
+  data.tar.gz: f02bb4c129449803a57b08bf9b7186af08a6d21edee1e68d18178b248b855e0b
 SHA512:
-  metadata.gz: 53630b742cd2723656b5d826a82f57042dcaab1e3259758aa6a71eac20ba387a23b3fb3905f0f36f3ce641b122229f366c9af1f029205b44cd82110f538a49cb
-  data.tar.gz: 199cd93e9898b7457d5e19b9cf1857b4d78ffc46777ca26778b094fa70ee02559dc90733e4e815eba74297ef9c066940792cef09e022801ef48c704e868fa8db
+  metadata.gz: cec0a3959cb3f093c7bb2b06213b4c7e983986466a115eb79e0b045a5b52fdda9797be41365db41a74d85bb52c384c147d68d3d84c503fef10c122bba7b72333
+  data.tar.gz: e99039606af3de0106e1ac26efaa78a06f09acbd9c2faaae9b4ca17339bdeb4830ba922bef84dc141317898556d09b1ec41cf1ebfebcce5e141d877c3fc34598

data/CHANGELOG.md

@@ -1,5 +1,73 @@
 # Changelog
 
+## 0.1.5
+
+Correctness/parity release. The native runtime architecture is unchanged, but
+the tokenizer/pooling contract is now pinned explicitly to `model2vec 0.9.0` +
+`tokenizers 0.23.1`, and CI contains an independent upstream boundary corpus
+instead of treating the Ruby `Reference` twin as proof of compatibility.
+
+### Fixed
+
+- **`[UNK]` is filtered before the embedding token cap.** `max_tokens: 512` now
+  means the first 512 usable (non-UNK, for `UNK_DROP`) ids, matching
+  `StaticModel` pooling semantics. Long OOV-heavy text can therefore scan past
+  512 raw ids instead of silently averaging fewer rows. `embed_with_stats` and
+  `embed_token_ids_with_stats` expose `pooled_count` alongside raw
+  `token_count` / `unk_count`.
+- **The supported Hugging Face AddedVocabulary subset is implemented.** The five
+  standard BERT special tokens (`[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]`)
+  are extracted literally before normalization when the source tokenizer marks
+  them `special: true, normalized: false`. Converter and runtime now share that
+  contract instead of merely whitelisting the metadata.
+- **BertNormalizer behavior matches the pinned Rust tokenizer at the known edges.**
+  The CJK Extension E range starts at `U+2B920` as in `tokenizers 0.23.1`, not
+  the Python `transformers` boundary `U+2B820`. The boundary corpus also locks
+  the actual `unicode_categories` 0.1.1 behavior for unassigned (`Cn`)
+  codepoints: despite a misleading comment in upstream `bert.rs`, that crate's
+  `is_other()` implementation checks Cc/Cf/Co, not Cn.
+- **Vocabulary hash lookup is bounded by the validated `max_probe`,** and model
+  loading rejects hash tables above the converter's 0.70 load factor.
+- **`load_model` model ids cannot escape the cache** through absolute paths,
+  `..`, empty path components, backslash traversal or NUL bytes.
+
+### Changed
+
+- **`.semb` format v3.** The header records the supported added-token mask and
+  the usable-id truncation policy. v2 files must be reconverted; this avoids a
+  runtime silently applying 0.1.5 semantics to a file whose tokenizer contract
+  was recorded by an older converter.
+- **Model2Vec's character pre-truncation heuristic is intentionally not copied.**
+  `max_tokens` means actual usable tokenizer ids. The external oracle records
+  rows where `StaticModel`'s `max_length * median_token_length` character cut
+  changes ids and reports them as an explicit semantic deviation rather than a
+  false parity failure/pass.
+- **The converter reads safetensors incrementally and accepts F32, F16 and BF16
+  embedding tensors.** `.semb` rows remain float32. `Format.write` also streams
+  sections and computes the checksum incrementally instead of assembling a
+  second whole-file Ruby string.
+- **Plain `require "static_embeddings"` loads runtime code only.** Converter,
+  safetensors, Unicode-table generation and the Ruby `Reference` are loaded only
+  by offline tooling/tests. The `static_embeddings convert` CLI explicitly loads
+  the converter before reading converter-specific defaults, so lazy loading does
+  not break the documented executable path.
+- **TLS token-id scratch keeps a bounded high-water mark.** It still reserves only
+  512 ids initially, but capacities grown by OOV-heavy usable-token truncation are
+  retained up to 8192 ids between calls. This removes grow/trim realloc churn on
+  realistic truncated inputs without pinning arbitrarily large per-thread buffers.
+
+### Verification
+
+- Added a deterministic **434-row boundary corpus** covering control/unassigned
+  boundaries, every supported CJK range edge ±1, punctuation classes, whitespace, combining
+  marks, AddedVocabulary, 511/512/513 boundaries, OOV ratios and long-prefix
+  cases.
+- CI now installs pinned `model2vec==0.9.0` / `tokenizers==0.23.1`, generates an
+  external oracle, and separately checks raw HF ids, usable ids, embedding
+  parity where the contracts are identical, and intentional character-precut
+  deviations. The existing 50k native-vs-Ruby differential fuzz remains as a
+  separate implementation-consistency layer.
+
 ## 0.1.4 (unreleased)
 
 Hot-path work in the C runtime. Token ids, the pooling contract and f16

data/README.md

@@ -32,8 +32,11 @@ never arbitrary HuggingFace models. A `.semb` file carries validated metadata, a
 BERT WordPiece tokenizer profile, an mmap-ready vocabulary lookup, float32
 embedding rows, and provenance plus a checksum.
 
-The only supported tokenizer profile is `BERT_WORDPIECE_V1`. The converter
-rejects unsupported tokenizer features rather than approximating them.
+The only supported tokenizer profile is `BERT_WORDPIECE_V1`, pinned for
+compatibility to `tokenizers 0.23.1`. The converter rejects unsupported
+tokenizer features rather than approximating them. `.semb` format v3 also
+records which of the five standard BERT added tokens are active. Files produced
+by 0.1.4 and earlier use format v2 and must be reconverted.
 
 ## Converting a model
 
@@ -57,6 +60,9 @@ model = StaticEmbeddings.load(ENV.fetch("EMBEDDING_MODEL"))   # explicit path
 Convert during image build or deploy preparation; production should only ever
 see `.semb` files.
 
+The offline converter accepts F32, F16 and BF16 safetensors embedding matrices;
+all three are converted to float32 rows in `.semb`.
+
 ### Verify once, not on every boot
 
 `load` does not check the SHA-256 by default: hashing a 135 MB file at boot
@@ -98,6 +104,7 @@ stats = model.embed_with_stats("postgres pipeline mode in Ruby")
 stats[:vector]
 stats[:token_count]
 stats[:unk_count]
+stats[:pooled_count]  # rows actually eligible for pooling
 stats[:truncated]
 
 ids         = model.tokenize("postgres pipeline mode in Ruby")
@@ -116,11 +123,19 @@ Output is a binary `String` of little-endian float32 values in row-major order.
 `embed_array` and `embed_batch_arrays` decode that into Ruby `Float` objects and
 exist for debugging and application code, not for the hot path.
 
-`embed_token_ids` pools ids you supply, applying the same `max_tokens`
-truncation as `embed`, so `embed_token_ids(model.tokenize(text))` equals
-`embed(text)`. Pass `max_tokens: false` to pool every id. It is for reusing a
-cached tokenization and for benchmarking pooling in isolation; it is not a
-faster path for ordinary text.
+`tokenize` exposes raw WordPiece ids, including `[UNK]`, and its own
+`max_tokens` cap is therefore a raw-id cap. Embedding has a different, upstream
+pooling rule: for `UNK_DROP`, `[UNK]` is removed first and `max_tokens` is then
+applied to usable ids. To cache tokenization without changing the embedding,
+cache it unbounded:
+
+```ruby
+ids = model.tokenize(text, max_tokens: false)
+model.embed_token_ids(ids) == model.embed(text)
+```
+
+`embed_token_ids` applies that same usable-id cap. This distinction matters only
+when `[UNK]` appears around the truncation boundary.
 
 ### `format: :f16`
 
@@ -175,13 +190,13 @@ model.dot_top_k(query, MATRIX, 10)          # lock-free, concurrent, GVL release
 
 ## Large inputs
 
-When truncation is active the runtime does not copy the whole string into C
-memory. It copies a leading slice sized from `max_tokens`, cut on a word
-boundary, and grows the budget only if that slice did not reach `max_tokens`.
-The result is identical to tokenizing the whole document, because WordPiece
-segmentation is word-local, but the bytes the tokenizer reads stop growing with
-input size: with `max_tokens: 512` the window is a few kilobytes whether the
-document is 10 KB or 3 MB.
+When truncation is active the runtime starts with a leading slice sized from
+`max_tokens`, cut on a word boundary, and grows the budget until it has reached
+`max_tokens` **usable** ids. For ordinary in-vocabulary text this keeps the
+tokenized prefix to a few kilobytes even when the document is megabytes long.
+For OOV-heavy input it may scan farther — or the whole string — because `[UNK]`
+no longer consumes the usable-token budget. That is required for the corrected
+UNK-before-truncate contract.
 
 That bounds the tokenizer, not the whole call. `embed` also has to establish
 that the Ruby `String` is valid UTF-8, and when Ruby has not computed the
@@ -273,22 +288,34 @@ subword splitting instead of hitting the vocabulary directly.
 
 ## Correctness contract
 
-The reference implementation is `model2vec.StaticModel`. The converter records
-its decisions in the `.semb` file: truncation at 512 tokens by default, applied
-after tokenization and before pooling; `[UNK]` tokens dropped; a zero vector for
-input with no usable tokens; L2 normalization when the source model requires it.
+0.1.5 separates contracts that 0.1.4 accidentally mixed together:
+
+- `tokenize` matches the pinned Hugging Face `tokenizers 0.23.1` raw
+  `BertNormalizer + BertPreTokenizer + WordPiece` ids, including the supported
+  `normalized: false` standard added tokens and `[UNK]`.
+- Embedding drops `[UNK]` when the model records `UNK_DROP`, then applies
+  `max_tokens` to the remaining usable ids, mean-pools, and performs source-model
+  L2 normalization.
+- The gem intentionally does **not** reproduce Model2Vec 0.9.0's character
+  pre-cut (`max_length * median_token_length`). Here `max_tokens` means actual
+  usable tokenizer ids. The oracle reports rows affected by this difference
+  separately instead of calling them parity.
 
-Token ids must match the reference exactly. Vectors are compared with tolerance,
-because floating-point addition order is not bit-stable across implementations:
+Where the execution contracts are the same, vectors are compared to
+`model2vec.StaticModel` with:
 
 ```text
 cosine >= 1 - 1e-6
 max_abs_diff < 1e-5
 ```
 
-`docs/MODEL_AUDIT.md` records the parity result, digests and edge-case
-behaviour for every trusted conversion, and the command to reproduce it. A new
-conversion is not trusted until it has its own record.
+The committed boundary generator currently produces 434 systematic rows. CI
+generates their oracle from pinned Python packages both on the deterministic
+fixture and on the pinned `potion-retrieval-32M` snapshot. The Ruby `Reference`
+fuzz is kept as a separate C-vs-Ruby implementation check and is not treated as
+proof of upstream compatibility. `docs/MODEL_AUDIT.md` records per-model audits;
+a corpus pass means exactly that corpus passed, not an exhaustive proof of all
+Unicode/tokenizer behavior.
 
 ## Development
 

data/Rakefile

@@ -44,7 +44,7 @@ task parity: :compile do
       Generate it first (needs Python and the model2vec package):
 
         python3 -m venv .venv-model2vec && . .venv-model2vec/bin/activate
-        pip install -U model2vec numpy
+        pip install "model2vec==0.9.0" "tokenizers==0.23.1" numpy
         python tools/model2vec_oracle.py /path/to/source-model --out #{oracle}
     MSG
   end

data/docs/ARCHITECTURE.md

@@ -33,7 +33,7 @@ in `se_internal.h`: the header fields are decoded little-endian explicitly, but
 the mmapped structures and the float matrix are read in native order, so a
 big-endian host is refused at load rather than silently misread.
 
-## `.semb` v2
+## `.semb` v3
 
 Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 
@@ -69,6 +69,7 @@ Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
 | 208–239 | 32 | two (offset, size) u64 pairs: `root_trie`, `continuation_trie` |
 | 240 | 32 | sha256 of the file with these 32 bytes zeroed |
 | 304 | 4 | max_probe observed in the vocabulary hash |
+| 308 | 4 | supported added-token bit mask |
 
 Sections, in fixed order: `vocab_strings`, `vocab_hash`, `embeddings`,
 `norm_tables`, `provenance`, `root_trie`, `continuation_trie`.
@@ -79,11 +80,12 @@ already has a slot for I8; there is no dead I8 path in C.
 ### Why the semantics live in the header
 
 `max_tokens_default`, `unk_policy`, `empty_policy` and the normalizer flags
-are properties of the *model*, not options the caller guesses. The reference
-implementation truncates at 512 tokens; a model card's `model_max_length` of
-1,000,000 is a different number entirely, and confusing them silently changes
-every vector for long chunks. Anything that can silently change output is a
-recorded field.
+are properties of the *model*, not options the caller guesses. The embedding contract caps the first 512 usable token ids by default;
+a model card's `model_max_length` of 1,000,000 is a different number entirely, and
+confusing them silently changes every vector for long chunks. Model2Vec 0.9.0
+also applies a character pre-cut before tokenization; this runtime intentionally
+does not copy that heuristic. Anything that can silently change output is a
+recorded field or an explicitly documented compatibility deviation.
 
 ## Vocabulary lookup and WordPiece trie
 
@@ -93,7 +95,7 @@ load factor ≤ 0.70, linear probing, FNV-1a with a fixed seed. The hash table i
 kept for validation, direct lookup and debugging.
 
 The hot WordPiece path no longer performs repeated hash lookups for every
-`end--` candidate. Format v2 also stores two converter-built, mmap-readable
+`end--` candidate. The format also stores two converter-built, mmap-readable
 tries inspired by double-array trie libraries such as libdatrie:
 
 - `root_trie` for tokens that can start a word;
@@ -135,8 +137,8 @@ Rejected alternatives:
 
 ## Unicode without a Unicode library
 
-`BertNormalizer` needs NFD, simple lowercase, and the Mn / P / C / Zs
-categories. Rather than vendoring utf8proc, the converter generates exactly
+`BertNormalizer` needs NFD, simple lowercase, and selected Unicode
+categories (Mn, punctuation, control/format/private-use, and whitespace). Rather than vendoring utf8proc, the converter generates exactly
 those tables from the Ruby VM's own Unicode data and writes them into the
 model file; the runtime binary-searches them.
 
@@ -146,15 +148,20 @@ maps `ß` to `ss`; `str::to_lowercase`, which HF uses, does not. A gem that
 picked the wrong one would produce plausible, slightly wrong vectors forever.
 
 The Ruby build that generated the tables is stamped into provenance, so a
-model file and its normalizer can never drift apart unnoticed.
+model file and its embedded normalizer tables can never drift apart unnoticed.
+That stamp is not, by itself, proof of Hugging Face parity: 0.1.5 therefore
+checks a systematic boundary corpus against pinned `tokenizers 0.23.1` in CI.
+One subtle example is `Cn`: `bert.rs` says `is_other()` includes unassigned
+codepoints, while the `unicode_categories 0.1.1` implementation actually tests
+Cc/Cf/Co. The runtime follows the executable upstream behavior, not the comment.
 
 ## The embedding kernel
 
 ```
-tokenize -> ids
-truncate to max_tokens          (before pooling — the reference contract)
+tokenize -> raw ids (including [UNK])
 drop [UNK] if unk_policy = DROP
-acc[j] += row[j] for each id, in token order
+truncate usable ids to max_tokens
+acc[j] += row[j] for each usable id, in token order
 divide by the number of pooled rows
 L2 normalize if the model says so
 ```
@@ -209,10 +216,11 @@ therefore serialised the process on work that the tokenizer would discard after
 
 `embed` and `embed_batch` instead copy a leading slice. The budget starts at
 `max_tokens * 16` bytes clamped to `[4096, 65536]`, and the slice is cut at a
-position where the tokenizer would have started a fresh word. A result is
-accepted only when that slice actually reached `max_tokens`; otherwise the
-budget doubles and the text is tokenized again. Growth is geometric, so the
-pathological case costs about twice the single-pass work.
+position where the tokenizer would have started a fresh word. A result is accepted only when that slice actually reached `max_tokens`
+usable ids; `[UNK]` does not consume the embedding budget. Otherwise the budget
+doubles and the text is tokenized again. Growth is geometric. OOV-heavy input
+may therefore reach the full document: deciding that fewer than `max_tokens`
+usable ids exist requires reading it.
 
 Three properties make this exact rather than approximate:
 
@@ -241,10 +249,11 @@ the predicate is Unicode-aware. Text with genuinely no legal cut inside the scan
 window - one enormous word, a run of combining marks - still falls back to a
 full copy, because there is nothing else that would be correct.
 
-`test/prefix_chunking_test.rb` sweeps every printable ASCII separator, checks
-CJK, NBSP and Unicode-punctuation documents against
-`embed_token_ids(tokenize(text))`, and asserts that quadrupling the input does
-not triple the cost.
+`test/prefix_chunking_test.rb` sweeps every printable ASCII separator and
+checks CJK, NBSP and Unicode-punctuation documents against unbounded raw
+tokenization followed by `embed_token_ids`. It asserts bounded prefix cost for
+in-vocabulary text; deliberately OOV-heavy input is checked for correctness, not
+flat cost.
 
 With `max_tokens: false` there is nothing to truncate and the whole input is
 copied.
@@ -299,10 +308,13 @@ interrupts a GVL-free region.
 No global mutable state exists in C. The model is immutable after load.
 Scratch buffers are reused per OS thread via `pthread_key` / `FlsAlloc`, and
 the destructor frees them when the thread exits — including the short-lived
-threads the fiber scheduler path creates per large call. On release the slot
-is trimmed back to the reserve sizes, so a single unlimited-`max_tokens`
-document does not pin its working set until the thread dies. A nested call on
-the same thread allocates a one-off heap scratch that is freed on release.
+threads the fiber scheduler path creates per large call. On release the
+codepoint/byte buffers are trimmed back to their reserve sizes. The token-id
+buffer keeps its observed high-water mark up to 8192 ids so repeated OOV-heavy
+usable-token truncation does not grow and shrink it on every call; larger
+capacities are trimmed back to that bound so one unlimited-`max_tokens`
+document cannot pin an arbitrary per-thread working set. A nested call on the
+same thread allocates a one-off heap scratch that is freed on release.
 
 ## Where this fits in a RAG pipeline
 
@@ -321,12 +333,11 @@ chunks -> .semb vectors --/
 
 ## Open questions before v1.0
 
-1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The 32M family
-   has a larger vocabulary than the bge-base tokenizer it was distilled from,
-   which means tokens were added somewhere. If they live in `added_tokens`
-   rather than `model.vocab`, HF matches them with a separate trie pass over
-   raw text that this runtime does not implement. The converter refuses such
-   models today; whether it has to is an audit question.
+1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The runtime now
+   implements the exact `normalized: false` extraction semantics needed by the
+   five standard BERT added tokens, but still rejects arbitrary AddedVocabulary,
+   BPE and Unigram profiles. Each source model still needs an audit before it is
+   treated as compatible.
 2. **Russian.** Distilling a multilingual teacher into a WordPiece vocabulary
    we control keeps the pure-C path, but skips the Tokenlearn pre-training
    that gives the published potion models their quality. That gap has to be

data/docs/LIMITATIONS.md

@@ -13,9 +13,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Only `.semb` files from this repository's converter.** Loading a
   HuggingFace directory at runtime is not supported and will not be. The
   converter is the only thing that reads third-party files, and it runs offline.
-- **Only the `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE, Unigram
-  and byte-level tokenizers are rejected at conversion time rather than
-  approximated.
+- **Only the pinned `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE,
+  Unigram and byte-level tokenizers are rejected. AddedVocabulary support is
+  deliberately limited to the five standard BERT special tokens with
+  `special: true`, `normalized: false` and no strip/single-word behavior.
 - **No model training, distillation or fine-tuning.** Produce the model with
   upstream `model2vec`, then convert it.
 - **No transformer inference.** Static embeddings have no attention and no
@@ -38,6 +39,10 @@ the entry says whether that is a decision or just unfinished work.
 - **Big-endian hosts are refused at load.** The header is decoded
   little-endian explicitly, but the mmapped structures and the float matrix are
   read in native order.
+- **Do not truncate/overwrite a mapped `.semb` in place.** On POSIX, shrinking
+  the inode underneath an existing mmap can raise `SIGBUS` in a worker. Deploy a
+  new file and atomically rename it over the path (`write .tmp` → `fsync` →
+  `rename`); existing workers keep their old inode mapping safely.
 
 ## Search
 
@@ -76,9 +81,10 @@ the entry says whether that is a decision or just unfinished work.
 - **`load` does not check the SHA-256.** Structural validation always runs, but
   a bit flip inside the matrix is only caught by `verify: true` or
   `StaticEmbeddings.verify`.
-- **Parity is per model and per runtime version.** A converted model is not
-  trusted until it has a record in `docs/MODEL_AUDIT.md`, and that record
-  expires when the tokenizer changes.
+- **Parity is per model, runtime, pinned upstream versions and corpus.** A
+  recorded pass means those rows passed; it is not an exhaustive proof of
+  tokenizer equivalence. The independent upstream CI and the Ruby implementation
+  fuzz intentionally test different things.
 
 ## Known open questions
 

data/docs/MODEL_AUDIT.md

@@ -1,27 +1,26 @@
 # Model Audit
 
-A converted model is trusted only after parity against upstream
-`model2vec.StaticModel` is recorded here. An audit record covers one `.semb`
-file **and** one runtime version: changing the tokenizer, normalizer, prefix
-window, pooling, output normalization or the oracle's corpus invalidates the
-runtime half of it even though the file bytes are untouched.
+A converted model is trusted only after a recorded comparison against an
+external upstream oracle. An audit record covers one source-model snapshot,
+one `.semb` format/runtime contract, pinned upstream package versions, and the
+specific corpus that was checked. A passing corpus is evidence for those rows;
+it is **not** an exhaustive proof that every Unicode string is equivalent.
+
+`StaticEmbeddings::Reference` remains useful for high-volume differential fuzz
+of the C implementation, but it is an implementation twin and is not counted as
+independent upstream evidence.
 
 ## potion-retrieval-32m
 
-| runtime | parity | note |
+| runtime | external result | note |
 |---|---|---|
-| 0.1.1 | `parity OK`, recorded below | superseded |
-| 0.1.2 | **not re-run** | superseded before release |
-| 0.1.3 | `parity OK` | tokenizer / pooling contract; same numbers as 0.1.4 |
-| 0.1.4 | `parity OK`, recorded below | current |
-
-0.1.2 changed `is_control()`, which changes token ids for any input containing
-`U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
-matches HuggingFace, because it is an implementation twin of the C runtime
-written in this repository. The 0.1.3 release candidate was checked against a
-fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
-characters, Unicode, OOV, long-word and truncation rows. 0.1.4 re-ran that
-oracle against the SIMD L2 runtime on the same snapshot and `.semb`.
+| 0.1.1 | 31-row corpus passed | superseded |
+| 0.1.2 | not re-run | superseded before release |
+| 0.1.3 | 31-row corpus passed | superseded |
+| 0.1.4 | 31-row corpus passed | historical result below; later review found uncovered boundary cases |
+| 0.1.5 | **434-row corpus passed** | format v3; CI `potion_audit` record below |
+
+### Historical 0.1.4 record
 
 Source model:
 
@@ -29,7 +28,6 @@ Source model:
 - Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
 - Python package: `model2vec 0.9.0` (`tokenizers 0.23.1`, `numpy 2.5.2`)
-- Oracle file: `tmp/model2vec_oracle.json`
 - Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
@@ -37,25 +35,12 @@ Source model:
 
 Converted `.semb`:
 
-- Path: `$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb`
 - Format version: `2`
 - Header size: `320`
 - Bytes: `135411608`
 - SHA256: `79e087863d2bab825779fd7de3574e5625542ef6a5ecad33fe681ea16d4b3ab0`
 
-Parity command:
-
-```bash
-python tools/model2vec_oracle.py \
-  ~/.cache/huggingface/hub/models--minishlab--potion-retrieval-32M/snapshots/6fc8051fab2a1e0ee76689cf08c853792ac285e7 \
-  --out tmp/model2vec_oracle.json
-
-bundle exec rake parity \
-  MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
-  ORACLE=tmp/model2vec_oracle.json
-```
-
-Parity result (0.1.4, 2026-08-31):
+Recorded result (2026-08-31):
 
 ```text
 rows=31
@@ -67,22 +52,70 @@ vector_failures=[]
 parity OK
 ```
 
-The printed `min_cosine` and `max_abs_all` are identical to the 0.1.3 record.
-Worst vector row is still idx=29 (`"word" + 400 spaces`, 80 800 bytes) at
-`max_abs=2.9802322e-07`. SIMD pairwise double L2 did not move the oracle
-agreement on this corpus.
-
-Decision:
-
-- Tokenization parity: pass
-- Vector parity: pass
-- Empty input behavior: pass
-- Whitespace behavior: pass
-- Unicode normalization behavior: pass
-- Unknown-word behavior: pass
-- Long input / truncation behavior: pass
-- DEL / control-character behavior: pass
-
-Accepted for runtime and benchmark use **under 0.1.4**. The `.semb` file is
-unchanged and its SHA256 still matches. The runtime half of the record was
-refreshed after the 0.1.4 L2 kernel change.
+The last line means **31/31 rows in that historical oracle passed**. It must not
+be read as a global tokenizer-equivalence claim. The corpus did not cover, for
+example, the corrected UNK-before-truncation ordering, standard AddedVocabulary
+literals, or the Rust/Python CJK Extension E boundary.
+
+## 0.1.5 record
+
+0.1.5 separates three contracts that the old oracle mixed together:
+
+1. raw Hugging Face `tokenizers 0.23.1` ids, including `[UNK]`;
+2. this runtime's usable-id embedding contract (`[UNK]` drop, then token cap);
+3. `model2vec.StaticModel` vectors where its character pre-cut does not change
+   the usable token sequence.
+
+Rows changed solely by Model2Vec's `max_length * median_token_length` character
+pre-cut are reported as intentional deviations rather than hidden inside a pass.
+A passing corpus is evidence for those 434 rows, not an exhaustive proof of
+every Unicode string.
+
+Recorded from GitHub Actions (`potion_audit`, Python 3.12.14, pinned
+`model2vec==0.9.0` / `tokenizers==0.23.1` / `numpy==2.5.2`):
+
+Source model:
+
+- Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
+- Oracle implementation: `model2vec.StaticModel.from_pretrained`
+- Oracle rows: `434` from `tools/parity_cases.py`
+- Oracle dimension: `512`
+- Oracle max length: `512`
+- Runtime at time of this record: `static_embeddings 0.1.5`
+
+Converted `.semb` (CI artifact; Unicode tables stamped from the Ubuntu Ruby that
+converted it, so the SHA is not expected to match a macOS local convert of the
+same snapshot):
+
+- Format version: `3`
+- Bytes: `135411800`
+- SHA256: `747231b5afbcb3b16bf2b04538f81d3a96be88a798982214b7cf01eddbdcf4eb`
+- `dim=512` `vocab=63091`
+
+Recorded result:
+
+```text
+rows=434
+vectors_checked=432
+intentional_character_pretruncate_deviations=2
+min_cosine=0.9999999999999989
+max_abs_all=1.4901161193847656e-08
+raw_token_id_failures=[]
+usable_token_id_failures=[]
+embed_invariant_failures=[]
+vector_failures=[]
+corpus parity OK (434/434); intentional Model2Vec character pre-truncation deviations are reported separately
+```
+
+The two intentional deviations are `long:sparse-whitespace` and
+`long:unknown-prefix`. Vectors were not required to match Model2Vec on those
+rows. The remaining 432 rows were inside `cosine >= 1 - 1e-6` and
+`max_abs <= 1e-5`.
+
+The preceding CI job `upstream_parity` ran the same 434-row corpus against the
+synthetic `tiny-wordpiece` fixture: `434/434`, all four failure lists empty,
+`vectors_checked=429`, `intentional_character_pretruncate_deviations=5`,
+`min_cosine=0.9999999999999988`, `max_abs_all=5.960464477539063e-08`. That job
+proves the fixture loads in `StaticModel.from_pretrained` and that the checker
+contracts hold; it is not a potion audit.

data/ext/static_embeddings/se_embed.c

@@ -197,7 +197,8 @@ se_status_t se_embed_ids(const se_model_t *model, se_scratch_t *sc, const uint32
 se_status_t se_embed_one(const se_model_t *model, se_scratch_t *sc, const uint8_t *input,
                          size_t input_len, uint32_t max_tokens, float *out, se_token_stats_t *stats,
                          se_error_t *err, volatile sig_atomic_t *cancelled) {
-    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, stats, err, cancelled);
+    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, SE_TOKEN_LIMIT_USABLE,
+                                 stats, err, cancelled);
     if (rc != SE_OK)
         return rc;
     return embed_ids_core(model, sc, sc->ids, stats->token_count, out, err, cancelled);