static_embeddings 0.1.4 → 1.5.6

data/docs/MODEL_AUDIT.md

@@ -1,27 +1,26 @@
 # Model Audit
 
-A converted model is trusted only after parity against upstream
-`model2vec.StaticModel` is recorded here. An audit record covers one `.semb`
-file **and** one runtime version: changing the tokenizer, normalizer, prefix
-window, pooling, output normalization or the oracle's corpus invalidates the
-runtime half of it even though the file bytes are untouched.
+A converted model is trusted only after a recorded comparison against an
+external upstream oracle. An audit record covers one source-model snapshot,
+one `.semb` format/runtime contract, pinned upstream package versions, and the
+specific corpus that was checked. A passing corpus is evidence for those rows;
+it is **not** an exhaustive proof that every Unicode string is equivalent.
+
+`StaticEmbeddings::Reference` remains useful for high-volume differential fuzz
+of the C implementation, but it is an implementation twin and is not counted as
+independent upstream evidence.
 
 ## potion-retrieval-32m
 
-| runtime | parity | note |
+| runtime | external result | note |
 |---|---|---|
-| 0.1.1 | `parity OK`, recorded below | superseded |
-| 0.1.2 | **not re-run** | superseded before release |
-| 0.1.3 | `parity OK` | tokenizer / pooling contract; same numbers as 0.1.4 |
-| 0.1.4 | `parity OK`, recorded below | current |
-
-0.1.2 changed `is_control()`, which changes token ids for any input containing
-`U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
-matches HuggingFace, because it is an implementation twin of the C runtime
-written in this repository. The 0.1.3 release candidate was checked against a
-fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
-characters, Unicode, OOV, long-word and truncation rows. 0.1.4 re-ran that
-oracle against the SIMD L2 runtime on the same snapshot and `.semb`.
+| 0.1.1 | 31-row corpus passed | superseded |
+| 0.1.2 | not re-run | superseded before release |
+| 0.1.3 | 31-row corpus passed | superseded |
+| 0.1.4 | 31-row corpus passed | historical result below; later review found uncovered boundary cases |
+| 0.1.5 | **434-row corpus passed** | format v3; CI `potion_audit` record below |
+
+### Historical 0.1.4 record
 
 Source model:
 
@@ -29,7 +28,6 @@ Source model:
 - Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
 - Python package: `model2vec 0.9.0` (`tokenizers 0.23.1`, `numpy 2.5.2`)
-- Oracle file: `tmp/model2vec_oracle.json`
 - Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
@@ -37,25 +35,12 @@ Source model:
 
 Converted `.semb`:
 
-- Path: `$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb`
 - Format version: `2`
 - Header size: `320`
 - Bytes: `135411608`
 - SHA256: `79e087863d2bab825779fd7de3574e5625542ef6a5ecad33fe681ea16d4b3ab0`
 
-Parity command:
-
-```bash
-python tools/model2vec_oracle.py \
-  ~/.cache/huggingface/hub/models--minishlab--potion-retrieval-32M/snapshots/6fc8051fab2a1e0ee76689cf08c853792ac285e7 \
-  --out tmp/model2vec_oracle.json
-
-bundle exec rake parity \
-  MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
-  ORACLE=tmp/model2vec_oracle.json
-```
-
-Parity result (0.1.4, 2026-08-31):
+Recorded result (2026-08-31):
 
 ```text
 rows=31
@@ -67,22 +52,180 @@ vector_failures=[]
 parity OK
 ```
 
-The printed `min_cosine` and `max_abs_all` are identical to the 0.1.3 record.
-Worst vector row is still idx=29 (`"word" + 400 spaces`, 80 800 bytes) at
-`max_abs=2.9802322e-07`. SIMD pairwise double L2 did not move the oracle
-agreement on this corpus.
-
-Decision:
-
-- Tokenization parity: pass
-- Vector parity: pass
-- Empty input behavior: pass
-- Whitespace behavior: pass
-- Unicode normalization behavior: pass
-- Unknown-word behavior: pass
-- Long input / truncation behavior: pass
-- DEL / control-character behavior: pass
-
-Accepted for runtime and benchmark use **under 0.1.4**. The `.semb` file is
-unchanged and its SHA256 still matches. The runtime half of the record was
-refreshed after the 0.1.4 L2 kernel change.
+The last line means **31/31 rows in that historical oracle passed**. It must not
+be read as a global tokenizer-equivalence claim. The corpus did not cover, for
+example, the corrected UNK-before-truncation ordering, standard AddedVocabulary
+literals, or the Rust/Python CJK Extension E boundary.
+
+## 0.1.5 record
+
+0.1.5 separates three contracts that the old oracle mixed together:
+
+1. raw Hugging Face `tokenizers 0.23.1` ids, including `[UNK]`;
+2. this runtime's usable-id embedding contract (`[UNK]` drop, then token cap);
+3. `model2vec.StaticModel` vectors where its character pre-cut does not change
+   the usable token sequence.
+
+Rows changed solely by Model2Vec's `max_length * median_token_length` character
+pre-cut are reported as intentional deviations rather than hidden inside a pass.
+A passing corpus is evidence for those 434 rows, not an exhaustive proof of
+every Unicode string.
+
+Recorded from GitHub Actions (`potion_audit`, Python 3.12.14, pinned
+`model2vec==0.9.0` / `tokenizers==0.23.1` / `numpy==2.5.2`):
+
+Source model:
+
+- Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
+- Oracle implementation: `model2vec.StaticModel.from_pretrained`
+- Oracle rows: `434` from `tools/parity_cases.py`
+- Oracle dimension: `512`
+- Oracle max length: `512`
+- Runtime at time of this record: `static_embeddings 0.1.5`
+
+Converted `.semb` (CI artifact; Unicode tables stamped from the Ubuntu Ruby that
+converted it, so the SHA is not expected to match a macOS local convert of the
+same snapshot):
+
+- Format version: `3`
+- Bytes: `135411800`
+- SHA256: `747231b5afbcb3b16bf2b04538f81d3a96be88a798982214b7cf01eddbdcf4eb`
+- `dim=512` `vocab=63091`
+
+Recorded result:
+
+```text
+rows=434
+vectors_checked=432
+intentional_character_pretruncate_deviations=2
+min_cosine=0.9999999999999989
+max_abs_all=1.4901161193847656e-08
+raw_token_id_failures=[]
+usable_token_id_failures=[]
+embed_invariant_failures=[]
+vector_failures=[]
+corpus parity OK (434/434); intentional Model2Vec character pre-truncation deviations are reported separately
+```
+
+The two intentional deviations are `long:sparse-whitespace` and
+`long:unknown-prefix`. Vectors were not required to match Model2Vec on those
+rows. The remaining 432 rows were inside `cosine >= 1 - 1e-6` and
+`max_abs <= 1e-5`.
+
+The preceding CI job `upstream_parity` ran the same 434-row corpus against the
+synthetic `tiny-wordpiece` fixture: `434/434`, all four failure lists empty,
+`vectors_checked=429`, `intentional_character_pretruncate_deviations=5`,
+`min_cosine=0.9999999999999988`, `max_abs_all=5.960464477539063e-08`. That job
+proves the fixture loads in `StaticModel.from_pretrained` and that the checker
+contracts hold; it is not a potion audit.
+
+## 1.5.6 Patch 1 proof
+
+Architecture: Model2Vec / Sentence Transformers import functions → immutable canonical data →
+existing WordPiece `.semb` v3 writer. C runtime unchanged.
+
+### potion-retrieval-32M reconvert
+
+Official snapshot `6fc8051fab2a1e0ee76689cf08c853792ac285e7` ships a Sentence
+Transformers `modules.json` (`StaticEmbedding` + `Normalize`). Detection uses
+`config.json` `model_type=model2vec` first, so this stays UNK_DROP / L2 / 512.
+
+New file `potion-retrieval-32m-reconvert.semb` vs existing
+`potion-retrieval-32m-v3.semb`: vocab/hash/embeddings/norm_tables/tries
+byte-identical. 10-text corpus `max_abs=0`. Provenance JSON differs (new keys).
+
+### static-retrieval-mrl-en-v1 vs SentenceTransformer.encode
+
+Oracle: `tools/st_oracle.py` + `tools/check_st_parity.rb`,
+`sentence-transformers 6.0.1`, `add_special_tokens=false`,
+`normalize_embeddings=false`, 436 rows including `all-unk:private-use` and
+`long:known-words-600`.
+
+```text
+1024: 436/436  min_cosine=0.999999999999999  max_abs_all=7.62939453125e-06
+512:  436/436  min_cosine=0.9999999999999989 max_abs_all=7.62939453125e-06
+```
+
+512 compared as prefix of the 1024-d encode vectors. Ruby Reference matched C
+separately. Diagnostic `StaticEmbedding.forward` matched encode on this model.
+
+### UNK and max_tokens on the real models
+
+- potion-retrieval-32M `🧬 🧬 🧬` → UNK ids dropped → zero vector; 600×`hello`
+  → `truncated=true pooled=512`.
+- static-retrieval-mrl-en-v1-1024 same emoji string → three UNK ids pooled
+  (non-zero, mean of UNK rows); 600×`hello` → `truncated=false pooled=600`.
+- Private-use codepoints are BertNormalizer `Co` and become empty, not UNK.
+
+## 1.5.6 Type A conversions
+
+These records prove the offline import layer maps extra WordPiece sources onto
+the existing runtime. They are **not** a replacement for the
+potion-retrieval-32m Python oracle above. `static-retrieval-mrl-en-v1` has the
+separate `SentenceTransformer.encode` oracle recorded above; the multilingual
+similarity artifact was checked against the Ruby `.semb` Reference twin and the
+small Russian retrieval fixture, not claimed as a general upstream retrieval
+benchmark.
+
+All conversions used `static_embeddings 1.5.6` on arm64-darwin24.
+
+### minishlab/potion-base-8M
+
+- Snapshot: `bf8b056651a2c21b8d2565580b8569da283cab23`
+- Layout: Model2Vec (`UNK_DROP`, L2, `max_tokens` 512)
+- `.semb`: `potion-base-8m.semb` `dim=256` `vocab=29528` `33346200` bytes
+  SHA256 `8abd8d1f26511959e14ca26ec4dcacad1d3db9b4b66b2e67683fb2d06074f490`
+- Native C matched Ruby Reference on short English/OOV/empty texts (`max_abs < 1e-5`)
+- `model2vec 0.9.0` / `tokenizers 0.23.1` oracle: 434/434, vectors_checked=432,
+  intentional character-pretruncate deviations=2, min_cosine=0.9999999999999989,
+  max_abs_all=2.98e-08
+
+### minishlab/potion-science-32M
+
+- Snapshot: `7366079845507de14a4330007cdfa01bb92bca52`
+- Layout: Model2Vec (`UNK_DROP`, L2, `max_tokens` 512)
+- `.semb`: `potion-science-32m.semb` `dim=256` `vocab=124428` `140353240` bytes
+  SHA256 `e968f93b89d59d9c63c6a8152495283d1138144e61ed5c08d306c21c00b7cdb1`
+- Native C matched Ruby Reference on English and Russian snippets (`max_abs < 1e-5`)
+- `model2vec 0.9.0` / `tokenizers 0.23.1` oracle: 434/434, vectors_checked=432,
+  intentional character-pretruncate deviations=2, min_cosine=0.9999999999999986,
+  max_abs_all=2.98e-08
+
+### sentence-transformers/static-retrieval-mrl-en-v1
+
+- Snapshot: `f60985c706f192d45d218078e49e5a8b6f15283a`
+- Layout: Sentence Transformers StaticEmbedding (`UNK_INCLUDE`, no L2, unlimited)
+- `static-retrieval-mrl-en-v1-1024.semb` `dim=1024` `vocab=30522` `128186264` bytes
+  SHA256 `f316bb07348503418ed0f6d897ddbf7b44422d2077ca4860518f31f328610946`
+- `static-retrieval-mrl-en-v1-512.semb` `--dimensions 512` `65677208` bytes
+  SHA256 `efbb43c14def793d0bbb42ef591a0951caed23a49cdd94dcf72312569d1b7cff`
+- Prefix of a 1024-d vector matched the 512-d artifact on in-vocabulary English
+  (valid because this source does not L2-normalize)
+- Native C matched Ruby Reference on short texts
+
+### sentence-transformers/static-similarity-mrl-multilingual-v1
+
+- Snapshot: `b68f4122911bcffcd6e1f695f2d99cd6788972d8`
+- Layout: Sentence Transformers StaticEmbedding (`UNK_INCLUDE`, no L2, unlimited)
+- `static-similarity-mrl-multilingual-v1-512.semb` `--dimensions 512` `vocab=105879`
+  `227983824` bytes SHA256 `8d9a63d20ee23468ae2fd2584ef3f0ec75e720e68410e416157aaf2b873e110e`
+- `static-similarity-mrl-multilingual-v1-256.semb` `--dimensions 256` `119563728` bytes
+  SHA256 `2e9e689bd3afd18ed50a77f0f8842b4b5a5be39bfad75e3fa24ad99b478de068`
+- Native C matched Ruby Reference on Russian text (`max_abs 9.1e-7`)
+
+This model is published as similarity, not retrieval. `tools/eval_retrieval.rb`
+on `test/fixtures/russian_faq_eval.json` (10 queries / 10 docs, cosine@10,
+arm64-darwin24, 1.5.6):
+
+| model | dim | MRR | nDCG@10 |
+|---|---:|---:|---:|
+| static-similarity-mrl-multilingual-v1-256 | 256 | 0.875 | 0.906 |
+| static-similarity-mrl-multilingual-v1-512 | 512 | 0.850 | 0.889 |
+| static-retrieval-mrl-en-v1-512 | 512 | 0.792 | 0.842 |
+| potion-retrieval-32m | 512 | 0.733 | 0.799 |
+| potion-base-8m | 256 | 0.712 | 0.780 |
+
+Hit@10 was 1.0 for every model on this tiny labeled set. That is a domain sanity
+check, not a public retrieval benchmark, and it does not make the similarity
+model a hard-coded Russian default.

data/ext/static_embeddings/se_embed.c

@@ -197,7 +197,8 @@ se_status_t se_embed_ids(const se_model_t *model, se_scratch_t *sc, const uint32
 se_status_t se_embed_one(const se_model_t *model, se_scratch_t *sc, const uint8_t *input,
                          size_t input_len, uint32_t max_tokens, float *out, se_token_stats_t *stats,
                          se_error_t *err, volatile sig_atomic_t *cancelled) {
-    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, stats, err, cancelled);
+    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, SE_TOKEN_LIMIT_USABLE,
+                                 stats, err, cancelled);
     if (rc != SE_OK)
         return rc;
     return embed_ids_core(model, sc, sc->ids, stats->token_count, out, err, cancelled);

data/ext/static_embeddings/se_format.c

@@ -102,7 +102,7 @@ int se_vocab_lookup_piece(const se_model_t *model, const uint8_t *prefix, size_t
     uint32_t pos = h & mask;
     uint32_t total_len = (uint32_t)(prefix_len + len);
 
-    for (uint32_t probe = 0; probe <= mask; probe++) {
+    for (uint32_t probe = 0; probe < model->meta.max_probe; probe++) {
         const se_vocab_slot_t *slot = &model->vocab_hash[pos];
         if (slot->token_id == SE_SLOT_EMPTY)
             return 0;
@@ -556,6 +556,31 @@ static se_status_t validate_vocab_hash(const se_model_t *model, struct section v
         se_error_set(err, SE_ERR_INVALID_FORMAT,
                      "[UNK] is not reachable through the vocabulary hash");
         rc = SE_ERR_INVALID_FORMAT;
+        goto done;
+    }
+
+    static const struct {
+        uint32_t bit;
+        const char *text;
+        uint32_t len;
+    } added[] = {
+        {SE_ADDED_PAD, "[PAD]", 5u}, {SE_ADDED_UNK, "[UNK]", 5u},   {SE_ADDED_CLS, "[CLS]", 5u},
+        {SE_ADDED_SEP, "[SEP]", 5u}, {SE_ADDED_MASK, "[MASK]", 6u},
+    };
+    const uint32_t ids[] = {m->pad_id, m->unk_id, m->cls_id, m->sep_id, m->mask_id};
+
+    for (size_t i = 0; i < SE_ARRAY_LEN(added); i++) {
+        if ((m->added_token_mask & added[i].bit) == 0)
+            continue;
+        uint32_t got = 0;
+        if (ids[i] >= m->vocab_size ||
+            !se_vocab_lookup(model, (const uint8_t *)added[i].text, added[i].len, &got) ||
+            got != ids[i]) {
+            se_error_set(err, SE_ERR_INVALID_FORMAT,
+                         "added token %s is not reachable at its recorded id", added[i].text);
+            rc = SE_ERR_INVALID_FORMAT;
+            goto done;
+        }
     }
 
 done:
@@ -646,6 +671,7 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
     m->subword_prefix_len = read_u32(base, SE_OFF_SUBWORD_PREFIX_LEN);
     m->max_token_chars = read_u32(base, SE_OFF_MAX_TOKEN_CHARS);
     m->max_probe = read_u32(base, SE_OFF_MAX_PROBE);
+    m->added_token_mask = read_u32(base, SE_OFF_ADDED_TOKEN_MASK);
     memcpy(m->subword_prefix, base + SE_OFF_SUBWORD_PREFIX, 8);
 
     if (m->tokenizer_type != SE_TOKENIZER_BERT_WORDPIECE_V1) {
@@ -669,7 +695,7 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
                      m->normalization_type);
         return SE_ERR_INVALID_FORMAT;
     }
-    if (m->truncation_policy != SE_TRUNCATE_IDS_BEFORE_POOLING) {
+    if (m->truncation_policy != SE_TRUNCATE_USABLE_IDS_BEFORE_POOLING) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "truncation policy %u is not supported",
                      m->truncation_policy);
         return SE_ERR_INVALID_FORMAT;
@@ -682,6 +708,11 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown UNK policy %u", m->unk_policy);
         return SE_ERR_INVALID_FORMAT;
     }
+    if ((m->added_token_mask & ~SE_ADDED_TOKEN_MASK_ALL) != 0) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown added-token mask bits 0x%x",
+                     m->added_token_mask & ~SE_ADDED_TOKEN_MASK_ALL);
+        return SE_ERR_INVALID_FORMAT;
+    }
     if (m->empty_policy != SE_EMPTY_ZERO_VECTOR && m->empty_policy != SE_EMPTY_RAISE) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown empty-input policy %u", m->empty_policy);
         return SE_ERR_INVALID_FORMAT;
@@ -704,8 +735,14 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
                      m->hash_table_size);
         return SE_ERR_INVALID_FORMAT;
     }
-    if (m->hash_table_size < m->vocab_size) {
-        se_error_set(err, SE_ERR_INVALID_FORMAT, "hash table smaller than vocabulary");
+    if (m->hash_table_size <= m->vocab_size ||
+        (uint64_t)m->vocab_size * 100u > (uint64_t)m->hash_table_size * 70u) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT,
+                     "hash table load factor exceeds the supported 0.70 maximum");
+        return SE_ERR_INVALID_FORMAT;
+    }
+    if (m->max_probe == 0 || m->max_probe > m->hash_table_size) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT, "invalid recorded max probe %u", m->max_probe);
         return SE_ERR_INVALID_FORMAT;
     }
     if (m->subword_prefix_len > 8) {

data/ext/static_embeddings/se_internal.h

@@ -8,7 +8,7 @@
 
 #define SE_MAGIC          "SEMBv1\0\0"
 #define SE_MAGIC_LEN      8
-#define SE_FORMAT_VERSION 2u
+#define SE_FORMAT_VERSION 3u
 #define SE_HEADER_SIZE    320u
 #define SE_ALIGNMENT      64u
 
@@ -18,7 +18,7 @@
 #define SE_NORMALIZATION_NONE          0u
 #define SE_NORMALIZATION_L2            1u
 
-#define SE_TRUNCATE_IDS_BEFORE_POOLING 1u
+#define SE_TRUNCATE_USABLE_IDS_BEFORE_POOLING 2u
 
 #define SE_UNK_INCLUDE 0u
 #define SE_UNK_DROP    1u
@@ -26,6 +26,14 @@
 #define SE_EMPTY_ZERO_VECTOR 0u
 #define SE_EMPTY_RAISE       1u
 
+#define SE_ADDED_PAD  (1u << 0)
+#define SE_ADDED_UNK  (1u << 1)
+#define SE_ADDED_CLS  (1u << 2)
+#define SE_ADDED_SEP  (1u << 3)
+#define SE_ADDED_MASK (1u << 4)
+#define SE_ADDED_TOKEN_MASK_ALL \
+    (SE_ADDED_PAD | SE_ADDED_UNK | SE_ADDED_CLS | SE_ADDED_SEP | SE_ADDED_MASK)
+
 #define SE_SLOT_EMPTY 0xFFFFFFFFu
 #define SE_SIZE_MAX   ((size_t)-1)
 
@@ -83,7 +91,8 @@ enum {
     SE_OFF_SEC_ROOT_TRIE = 208,
     SE_OFF_SEC_CONT_TRIE = 224,
     SE_OFF_CHECKSUM = 240,
-    SE_OFF_MAX_PROBE = 304
+    SE_OFF_MAX_PROBE = 304,
+    SE_OFF_ADDED_TOKEN_MASK = 308
 };
 
 typedef struct {
@@ -163,6 +172,7 @@ typedef struct {
     uint32_t max_input_chars_per_word;
     uint32_t max_token_chars;
     uint32_t max_probe;
+    uint32_t added_token_mask;
     uint32_t pad_id;
     uint32_t unk_id;
     uint32_t cls_id;
@@ -401,9 +411,11 @@ typedef struct {
     uint32_t truncated;
 } se_token_stats_t;
 
+typedef enum { SE_TOKEN_LIMIT_RAW = 0, SE_TOKEN_LIMIT_USABLE = 1 } se_token_limit_t;
+
 se_status_t se_tokenize(const se_model_t *model, se_scratch_t *scratch, const uint8_t *input,
-                        size_t input_len, uint32_t max_tokens, se_token_stats_t *stats,
-                        se_error_t *err, volatile sig_atomic_t *cancelled);
+                        size_t input_len, uint32_t max_tokens, se_token_limit_t limit_mode,
+                        se_token_stats_t *stats, se_error_t *err, volatile sig_atomic_t *cancelled);
 
 se_status_t se_embed_one(const se_model_t *model, se_scratch_t *scratch, const uint8_t *input,
                          size_t input_len, uint32_t max_tokens, float *out, se_token_stats_t *stats,