static_embeddings 0.1.3 → 0.1.5

data/docs/MODEL_AUDIT.md

@@ -1,58 +1,46 @@
 # Model Audit
 
-A converted model is trusted only after parity against upstream
-`model2vec.StaticModel` is recorded here. An audit record covers one `.semb`
-file **and** one runtime version: changing the tokenizer, normalizer, prefix
-window, pooling, output normalization or the oracle's corpus invalidates the
-runtime half of it even though the file bytes are untouched.
+A converted model is trusted only after a recorded comparison against an
+external upstream oracle. An audit record covers one source-model snapshot,
+one `.semb` format/runtime contract, pinned upstream package versions, and the
+specific corpus that was checked. A passing corpus is evidence for those rows;
+it is **not** an exhaustive proof that every Unicode string is equivalent.
+
+`StaticEmbeddings::Reference` remains useful for high-volume differential fuzz
+of the C implementation, but it is an implementation twin and is not counted as
+independent upstream evidence.
 
 ## potion-retrieval-32m
 
-| runtime | parity | note |
+| runtime | external result | note |
 |---|---|---|
-| 0.1.1 | `parity OK`, recorded below | superseded |
-| 0.1.2 | **not re-run** | superseded before release |
-| 0.1.3 | `parity OK`, recorded below | current |
+| 0.1.1 | 31-row corpus passed | superseded |
+| 0.1.2 | not re-run | superseded before release |
+| 0.1.3 | 31-row corpus passed | superseded |
+| 0.1.4 | 31-row corpus passed | historical result below; later review found uncovered boundary cases |
+| 0.1.5 | **434-row corpus passed** | format v3; CI `potion_audit` record below |
 
-0.1.2 changed `is_control()`, which changes token ids for any input containing
-`U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
-matches HuggingFace, because it is an implementation twin of the C runtime
-written in this repository. The 0.1.3 release candidate was checked against a
-fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
-characters, Unicode, OOV, long-word and truncation rows.
+### Historical 0.1.4 record
 
 Source model:
 
 - Hugging Face repository: `minishlab/potion-retrieval-32M`
 - Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
-- Python package: `model2vec 0.9.0`
-- Oracle file: `tmp/model2vec_oracle.json`
+- Python package: `model2vec 0.9.0` (`tokenizers 0.23.1`, `numpy 2.5.2`)
 - Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
-- Runtime at time of this record: `static_embeddings 0.1.3`
+- Runtime at time of this record: `static_embeddings 0.1.4`
 
 Converted `.semb`:
 
-- Path: `$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb`
 - Format version: `2`
 - Header size: `320`
 - Bytes: `135411608`
 - SHA256: `79e087863d2bab825779fd7de3574e5625542ef6a5ecad33fe681ea16d4b3ab0`
 
-Parity command:
-
-```bash
-python tools/model2vec_oracle.py minishlab/potion-retrieval-32M \
-  --out tmp/model2vec_oracle.json
-
-bundle exec rake parity \
-  MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
-  ORACLE=tmp/model2vec_oracle.json
-```
-
-Parity result:
+Recorded result (2026-08-31):
 
 ```text
 rows=31
@@ -64,17 +52,70 @@ vector_failures=[]
 parity OK
 ```
 
-Decision:
+The last line means **31/31 rows in that historical oracle passed**. It must not
+be read as a global tokenizer-equivalence claim. The corpus did not cover, for
+example, the corrected UNK-before-truncation ordering, standard AddedVocabulary
+literals, or the Rust/Python CJK Extension E boundary.
+
+## 0.1.5 record
+
+0.1.5 separates three contracts that the old oracle mixed together:
+
+1. raw Hugging Face `tokenizers 0.23.1` ids, including `[UNK]`;
+2. this runtime's usable-id embedding contract (`[UNK]` drop, then token cap);
+3. `model2vec.StaticModel` vectors where its character pre-cut does not change
+   the usable token sequence.
+
+Rows changed solely by Model2Vec's `max_length * median_token_length` character
+pre-cut are reported as intentional deviations rather than hidden inside a pass.
+A passing corpus is evidence for those 434 rows, not an exhaustive proof of
+every Unicode string.
+
+Recorded from GitHub Actions (`potion_audit`, Python 3.12.14, pinned
+`model2vec==0.9.0` / `tokenizers==0.23.1` / `numpy==2.5.2`):
+
+Source model:
+
+- Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
+- Oracle implementation: `model2vec.StaticModel.from_pretrained`
+- Oracle rows: `434` from `tools/parity_cases.py`
+- Oracle dimension: `512`
+- Oracle max length: `512`
+- Runtime at time of this record: `static_embeddings 0.1.5`
+
+Converted `.semb` (CI artifact; Unicode tables stamped from the Ubuntu Ruby that
+converted it, so the SHA is not expected to match a macOS local convert of the
+same snapshot):
+
+- Format version: `3`
+- Bytes: `135411800`
+- SHA256: `747231b5afbcb3b16bf2b04538f81d3a96be88a798982214b7cf01eddbdcf4eb`
+- `dim=512` `vocab=63091`
+
+Recorded result:
 
-- Tokenization parity: pass
-- Vector parity: pass
-- Empty input behavior: pass
-- Whitespace behavior: pass
-- Unicode normalization behavior: pass
-- Unknown-word behavior: pass
-- Long input / truncation behavior: pass
-- DEL / control-character behavior: pass
+```text
+rows=434
+vectors_checked=432
+intentional_character_pretruncate_deviations=2
+min_cosine=0.9999999999999989
+max_abs_all=1.4901161193847656e-08
+raw_token_id_failures=[]
+usable_token_id_failures=[]
+embed_invariant_failures=[]
+vector_failures=[]
+corpus parity OK (434/434); intentional Model2Vec character pre-truncation deviations are reported separately
+```
 
-Accepted for runtime and benchmark use **under 0.1.3**. The `.semb` file is
-unchanged and its SHA256 still matches; the runtime half of the record was
-refreshed after the DEL fix.
+The two intentional deviations are `long:sparse-whitespace` and
+`long:unknown-prefix`. Vectors were not required to match Model2Vec on those
+rows. The remaining 432 rows were inside `cosine >= 1 - 1e-6` and
+`max_abs <= 1e-5`.
+
+The preceding CI job `upstream_parity` ran the same 434-row corpus against the
+synthetic `tiny-wordpiece` fixture: `434/434`, all four failure lists empty,
+`vectors_checked=429`, `intentional_character_pretruncate_deviations=5`,
+`min_cosine=0.9999999999999988`, `max_abs_all=5.960464477539063e-08`. That job
+proves the fixture loads in `StaticModel.from_pretrained` and that the checker
+contracts hold; it is not a potion audit.

data/docs/PERFORMANCE.md

@@ -22,33 +22,42 @@ ruby tools/benchmark.rb [model.semb]
 
 ## Where the time actually goes
 
-Tokenization dominates; pooling is a handful of row reads and adds. That is why
-the tokenizer is the file to profile, and why SIMD in the pooling loop is not
-where the wins are — that loop is memory-bound on random row lookups and the
-compiler already vectorises it.
-
-Out-of-vocabulary text is the main cliff. In-vocabulary words hit the
-vocabulary hash directly; everything else falls through to trie-driven subword
-splitting, which on a synthetic all-OOV corpus measured several times the
-per-text cost. Non-English input on an English model pays this on top of the
-`[UNK]` quality problem.
+Which file to profile depends on the model and the text. On a short-dimension
+demo model, or on synthetic all-OOV input, tokenization dominates. On
+`potion-retrieval-32m` (`dim=512`) with in-vocabulary English, pooling is the
+larger share: each token is a random 2 KiB row add, and before 0.1.4 the scalar
+L2 pass showed up as about 19% of an ASCII `embed_batch` sample tree. SIMD in
+that L2 slot was a real win; another unroll of the add loop was not — it is
+memory-bound on those gathers, and the compiler already vectorises it.
+
+Out-of-vocabulary text is the remaining tokenizer cliff. In-vocabulary words
+hit the vocabulary hash directly; everything else falls through to trie-driven
+subword splitting, which on a synthetic all-OOV corpus measured several times
+the per-text cost. Non-English input on an English model pays this on top of
+the `[UNK]` quality problem. Quote `mean_tokens_per_text` and `unk_ratio` from
+the batch samples before comparing texts/s across modes: hashes are slower
+than ASCII on this corpus mostly because they emit ~73 tokens/text against
+~31, not only because they miss the hash.
 
 ## `f16` is a storage trade-off
 
-`format: :f16` halves the bytes a top-k scan streams and doubles the decode
-work. Which one wins is a property of the machine, and this repository's own
-sample runs disagree with each other — same corpus, same `k`, native decode
-kernel in both cases:
+`format: :f16` halves the bytes a top-k scan streams and costs a decode per
+row. Which one wins is a property of the machine and of the decode kernel, and
+this repository's own sample runs disagree with each other — same corpus, same
+`k`, native decode kernel in both cases:
 
 ```text
                      f32        f16
-x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster
-M1 Pro, neon-fp16  2.29 ms    2.88 ms    f16 1.3x slower
+x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster   (F16C kernel, unchanged)
+M1 Pro, neon-fp16  2.36 ms    1.81 ms    f16 1.3x faster   (0.1.4, 16-wide NEON)
 ```
 
-Neither ratio transfers. Confirm `StaticEmbeddings.simd_backend`, then measure
-on the hardware you will run on. On the lookup-table fallback `f16` is usually
-slower than `f32`.
+The M1 Pro row in 0.1.3 was 2.29 / 2.88 ms, f16 1.3x *slower*, with an 8-wide
+decode. Neither ratio transfers. Confirm `StaticEmbeddings.simd_backend`, then
+measure on the hardware you will run on. On the lookup-table fallback `f16` is
+usually slower than `f32`. `embed_batch(format: :f16)` still encodes the
+returned blob with the scalar half converter, so a batch that is not a top-k
+scan can still be slower than `f32` on the same machine.
 
 ## C allocation counters
 

data/ext/static_embeddings/se_embed.c

@@ -8,6 +8,7 @@
 #define SE_HAVE_NEON 1
 #elif defined(__SSE__)
 #include <xmmintrin.h>
+#include <emmintrin.h>
 #define SE_HAVE_SSE 1
 #endif
 
@@ -71,15 +72,49 @@ static void scale_copy(float *out, const float *acc, uint32_t dim, float inv) {
         out[j] = acc[j] * inv;
 }
 
-void se_l2_normalize(float *vec, uint32_t dim) {
+static double sum_sq_d(const float *vec, uint32_t dim) {
+    uint32_t j = 0;
     double sum = 0.0;
-    for (uint32_t j = 0; j < dim; j++)
+#if defined(SE_HAVE_NEON) && defined(__aarch64__)
+    float64x2_t s0 = vdupq_n_f64(0.0);
+    float64x2_t s1 = vdupq_n_f64(0.0);
+    for (; j + 7 < dim; j += 8) {
+        float32x4_t a = vld1q_f32(vec + j);
+        float32x4_t b = vld1q_f32(vec + j + 4);
+        float64x2_t a0 = vcvt_f64_f32(vget_low_f32(a));
+        float64x2_t a1 = vcvt_f64_f32(vget_high_f32(a));
+        float64x2_t b0 = vcvt_f64_f32(vget_low_f32(b));
+        float64x2_t b1 = vcvt_f64_f32(vget_high_f32(b));
+        s0 = vaddq_f64(s0, vmulq_f64(a0, a0));
+        s1 = vaddq_f64(s1, vmulq_f64(a1, a1));
+        s0 = vaddq_f64(s0, vmulq_f64(b0, b0));
+        s1 = vaddq_f64(s1, vmulq_f64(b1, b1));
+    }
+    sum = vaddvq_f64(vaddq_f64(s0, s1));
+#elif defined(SE_HAVE_SSE)
+    __m128d s0 = _mm_setzero_pd();
+    __m128d s1 = _mm_setzero_pd();
+    for (; j + 3 < dim; j += 4) {
+        __m128 v = _mm_loadu_ps(vec + j);
+        __m128d lo = _mm_cvtps_pd(v);
+        __m128d hi = _mm_cvtps_pd(_mm_movehl_ps(v, v));
+        s0 = _mm_add_pd(s0, _mm_mul_pd(lo, lo));
+        s1 = _mm_add_pd(s1, _mm_mul_pd(hi, hi));
+    }
+    double tmp[2];
+    _mm_storeu_pd(tmp, _mm_add_pd(s0, s1));
+    sum = tmp[0] + tmp[1];
+#endif
+    for (; j < dim; j++)
         sum += (double)vec[j] * (double)vec[j];
+    return sum;
+}
+
+void se_l2_normalize(float *vec, uint32_t dim) {
+    double sum = sum_sq_d(vec, dim);
     if (sum <= 0.0)
         return;
-    float inv = (float)(1.0 / sqrt(sum));
-    for (uint32_t j = 0; j < dim; j++)
-        vec[j] *= inv;
+    scale_copy(vec, vec, dim, (float)(1.0 / sqrt(sum)));
 }
 
 static se_status_t embed_ids_core(const se_model_t *model, se_scratch_t *sc, const uint32_t *ids,
@@ -128,8 +163,12 @@ static se_status_t embed_ids_core(const se_model_t *model, se_scratch_t *sc, con
     }
 
     if (model->meta.normalization_type == SE_NORMALIZATION_L2) {
-        scale_copy(out, acc, dim, 1.0f);
-        se_l2_normalize(out, dim);
+        double sum = sum_sq_d(acc, dim);
+        if (sum <= 0.0) {
+            memset(out, 0, (size_t)dim * sizeof(float));
+            return SE_OK;
+        }
+        scale_copy(out, acc, dim, (float)(1.0 / sqrt(sum)));
     } else {
         const float inv = 1.0f / (float)used;
         scale_copy(out, acc, dim, inv);
@@ -158,7 +197,8 @@ se_status_t se_embed_ids(const se_model_t *model, se_scratch_t *sc, const uint32
 se_status_t se_embed_one(const se_model_t *model, se_scratch_t *sc, const uint8_t *input,
                          size_t input_len, uint32_t max_tokens, float *out, se_token_stats_t *stats,
                          se_error_t *err, volatile sig_atomic_t *cancelled) {
-    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, stats, err, cancelled);
+    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, SE_TOKEN_LIMIT_USABLE,
+                                 stats, err, cancelled);
     if (rc != SE_OK)
         return rc;
     return embed_ids_core(model, sc, sc->ids, stats->token_count, out, err, cancelled);

data/ext/static_embeddings/se_f16.c

@@ -221,17 +221,31 @@ static float dot_product_f16_neon(const float *q, const uint8_t *row, size_t dim
     size_t j = 0;
     float32x4_t a0 = vdupq_n_f32(0.0f);
     float32x4_t a1 = vdupq_n_f32(0.0f);
+    float32x4_t a2 = vdupq_n_f32(0.0f);
+    float32x4_t a3 = vdupq_n_f32(0.0f);
+    for (; j + 15 < dim; j += 16) {
+        float16x8_t h0 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + j * 2)));
+        float16x8_t h1 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + (j + 8) * 2)));
+        float32x4_t r0 = vcvt_f32_f16(vget_low_f16(h0));
+        float32x4_t r1 = vcvt_f32_f16(vget_high_f16(h0));
+        float32x4_t r2 = vcvt_f32_f16(vget_low_f16(h1));
+        float32x4_t r3 = vcvt_f32_f16(vget_high_f16(h1));
+        a0 = vmlaq_f32(a0, vld1q_f32(q + j), r0);
+        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), r1);
+        a2 = vmlaq_f32(a2, vld1q_f32(q + j + 8), r2);
+        a3 = vmlaq_f32(a3, vld1q_f32(q + j + 12), r3);
+    }
     for (; j + 7 < dim; j += 8) {
         float16x4_t h0 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + j * 2)));
         float16x4_t h1 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + (j + 4) * 2)));
-        float32x4_t r0 = vcvt_f32_f16(h0);
-        float32x4_t r1 = vcvt_f32_f16(h1);
-        a0 = vmlaq_f32(a0, vld1q_f32(q + j), r0);
-        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), r1);
+        a0 = vmlaq_f32(a0, vld1q_f32(q + j), vcvt_f32_f16(h0));
+        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), vcvt_f32_f16(h1));
     }
-    float32x4_t sumv = vaddq_f32(a0, a1);
+    float32x4_t sumv = vaddq_f32(vaddq_f32(a0, a1), vaddq_f32(a2, a3));
     float dot = vaddvq_f32(sumv);
     for (; j < dim; j++)
         dot += q[j] * se_f16_bits_to_float(read_u16le(row + j * 2));
@@ -243,8 +257,30 @@ static float dot_and_row_sq_f16_neon(const float *q, const uint8_t *row, size_t
     size_t j = 0;
     float32x4_t d0 = vdupq_n_f32(0.0f);
     float32x4_t d1 = vdupq_n_f32(0.0f);
+    float32x4_t d2 = vdupq_n_f32(0.0f);
+    float32x4_t d3 = vdupq_n_f32(0.0f);
     float32x4_t s0 = vdupq_n_f32(0.0f);
     float32x4_t s1 = vdupq_n_f32(0.0f);
+    float32x4_t s2 = vdupq_n_f32(0.0f);
+    float32x4_t s3 = vdupq_n_f32(0.0f);
+    for (; j + 15 < dim; j += 16) {
+        float16x8_t h0 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + j * 2)));
+        float16x8_t h1 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + (j + 8) * 2)));
+        float32x4_t r0 = vcvt_f32_f16(vget_low_f16(h0));
+        float32x4_t r1 = vcvt_f32_f16(vget_high_f16(h0));
+        float32x4_t r2 = vcvt_f32_f16(vget_low_f16(h1));
+        float32x4_t r3 = vcvt_f32_f16(vget_high_f16(h1));
+        d0 = vmlaq_f32(d0, vld1q_f32(q + j), r0);
+        d1 = vmlaq_f32(d1, vld1q_f32(q + j + 4), r1);
+        d2 = vmlaq_f32(d2, vld1q_f32(q + j + 8), r2);
+        d3 = vmlaq_f32(d3, vld1q_f32(q + j + 12), r3);
+        s0 = vmlaq_f32(s0, r0, r0);
+        s1 = vmlaq_f32(s1, r1, r1);
+        s2 = vmlaq_f32(s2, r2, r2);
+        s3 = vmlaq_f32(s3, r3, r3);
+    }
     for (; j + 7 < dim; j += 8) {
         float16x4_t h0 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + j * 2)));
@@ -257,8 +293,8 @@ static float dot_and_row_sq_f16_neon(const float *q, const uint8_t *row, size_t
         s0 = vmlaq_f32(s0, r0, r0);
         s1 = vmlaq_f32(s1, r1, r1);
     }
-    float dot = vaddvq_f32(vaddq_f32(d0, d1));
-    float row_sq = vaddvq_f32(vaddq_f32(s0, s1));
+    float dot = vaddvq_f32(vaddq_f32(vaddq_f32(d0, d1), vaddq_f32(d2, d3)));
+    float row_sq = vaddvq_f32(vaddq_f32(vaddq_f32(s0, s1), vaddq_f32(s2, s3)));
     for (; j < dim; j++) {
         float r = se_f16_bits_to_float(read_u16le(row + j * 2));
         dot += q[j] * r;

data/ext/static_embeddings/se_format.c

@@ -61,6 +61,34 @@ static uint32_t hash_bytes_continue(uint32_t h, const uint8_t *data, size_t len)
     return h;
 }
 
+static int se_memeq(const uint8_t *a, const uint8_t *b, size_t n) {
+    while (n >= 8) {
+        uint64_t ua, ub;
+        memcpy(&ua, a, 8);
+        memcpy(&ub, b, 8);
+        if (ua != ub)
+            return 0;
+        a += 8;
+        b += 8;
+        n -= 8;
+    }
+    if (n >= 4) {
+        uint32_t ua, ub;
+        memcpy(&ua, a, 4);
+        memcpy(&ub, b, 4);
+        if (ua != ub)
+            return 0;
+        a += 4;
+        b += 4;
+        n -= 4;
+    }
+    while (n--) {
+        if (*a++ != *b++)
+            return 0;
+    }
+    return 1;
+}
+
 int se_vocab_lookup_piece(const se_model_t *model, const uint8_t *prefix, size_t prefix_len,
                           const uint8_t *bytes, size_t len, uint32_t *id_out) {
     if (prefix_len > UINT32_MAX || len > UINT32_MAX || prefix_len > UINT32_MAX - len)
@@ -74,14 +102,14 @@ int se_vocab_lookup_piece(const se_model_t *model, const uint8_t *prefix, size_t
     uint32_t pos = h & mask;
     uint32_t total_len = (uint32_t)(prefix_len + len);
 
-    for (uint32_t probe = 0; probe <= mask; probe++) {
+    for (uint32_t probe = 0; probe < model->meta.max_probe; probe++) {
         const se_vocab_slot_t *slot = &model->vocab_hash[pos];
         if (slot->token_id == SE_SLOT_EMPTY)
             return 0;
         if (slot->hash == h && slot->str_len == total_len) {
-            const char *token = model->vocab_strings + slot->str_off;
-            if ((prefix_len == 0 || memcmp(token, prefix, prefix_len) == 0) &&
-                (len == 0 || memcmp(token + prefix_len, bytes, len) == 0)) {
+            const uint8_t *token = (const uint8_t *)(model->vocab_strings + slot->str_off);
+            if ((prefix_len == 0 || se_memeq(token, prefix, prefix_len)) &&
+                (len == 0 || se_memeq(token + prefix_len, bytes, len))) {
                 *id_out = slot->token_id;
                 return 1;
             }
@@ -103,6 +131,17 @@ static inline uint32_t trie_find_child(const se_trie_t *trie, const se_trie_node
     if (count == 0)
         return SE_SLOT_EMPTY;
 
+    if (count == 1)
+        return trie->edges[start].byte == byte ? trie->edges[start].child : SE_SLOT_EMPTY;
+
+    if (count == 2) {
+        if (trie->edges[start].byte == byte)
+            return trie->edges[start].child;
+        if (trie->edges[start + 1u].byte == byte)
+            return trie->edges[start + 1u].child;
+        return SE_SLOT_EMPTY;
+    }
+
     if (count <= 8) {
         for (uint32_t i = 0; i < count; i++) {
             const se_trie_edge_t *edge = &trie->edges[start + i];
@@ -342,6 +381,37 @@ static se_status_t validate_norm_ranges(const se_range_t *ranges, uint32_t count
     return SE_OK;
 }
 
+static void fill_map256(const se_map_entry_t **out, const se_map_entry_t *entries, uint32_t count) {
+    memset(out, 0, 256 * sizeof(*out));
+    for (uint32_t i = 0; i < count; i++) {
+        if (entries[i].cp < 256)
+            out[entries[i].cp] = &entries[i];
+    }
+}
+
+static void fill_range256(uint8_t *out, const se_range_t *ranges, uint32_t count) {
+    memset(out, 0, 256);
+    for (uint32_t i = 0; i < count; i++) {
+        uint32_t lo = ranges[i].lo;
+        uint32_t hi = ranges[i].hi;
+        if (lo > 255)
+            continue;
+        if (hi > 255)
+            hi = 255;
+        for (uint32_t cp = lo; cp <= hi; cp++)
+            out[cp] = 1;
+    }
+}
+
+static void prepare_norm_fast_tables(se_model_t *model) {
+    fill_map256(model->norm.lower256, model->norm.lower, model->norm.lower_count);
+    fill_map256(model->norm.nfd256, model->norm.nfd, model->norm.nfd_count);
+    fill_range256(model->norm.mn256, model->norm.mn, model->norm.mn_count);
+    fill_range256(model->norm.punct256, model->norm.punct, model->norm.punct_count);
+    fill_range256(model->norm.control256, model->norm.control, model->norm.control_count);
+    fill_range256(model->norm.whitespace256, model->norm.whitespace, model->norm.whitespace_count);
+}
+
 static se_status_t parse_norm_tables(se_model_t *model, const uint8_t *base, struct section sec,
                                      se_error_t *err) {
     memset(&model->norm, 0, sizeof(model->norm));
@@ -417,8 +487,13 @@ static se_status_t parse_norm_tables(se_model_t *model, const uint8_t *base, str
     rc = validate_norm_ranges(model->norm.control, model->norm.control_count, "control", err);
     if (rc != SE_OK)
         return rc;
-    return validate_norm_ranges(model->norm.whitespace, model->norm.whitespace_count, "whitespace",
-                                err);
+    rc = validate_norm_ranges(model->norm.whitespace, model->norm.whitespace_count, "whitespace",
+                              err);
+    if (rc != SE_OK)
+        return rc;
+
+    prepare_norm_fast_tables(model);
+    return SE_OK;
 }
 
 static se_status_t validate_vocab_hash(const se_model_t *model, struct section vocab_strings,
@@ -481,6 +556,31 @@ static se_status_t validate_vocab_hash(const se_model_t *model, struct section v
         se_error_set(err, SE_ERR_INVALID_FORMAT,
                      "[UNK] is not reachable through the vocabulary hash");
         rc = SE_ERR_INVALID_FORMAT;
+        goto done;
+    }
+
+    static const struct {
+        uint32_t bit;
+        const char *text;
+        uint32_t len;
+    } added[] = {
+        {SE_ADDED_PAD, "[PAD]", 5u}, {SE_ADDED_UNK, "[UNK]", 5u},   {SE_ADDED_CLS, "[CLS]", 5u},
+        {SE_ADDED_SEP, "[SEP]", 5u}, {SE_ADDED_MASK, "[MASK]", 6u},
+    };
+    const uint32_t ids[] = {m->pad_id, m->unk_id, m->cls_id, m->sep_id, m->mask_id};
+
+    for (size_t i = 0; i < SE_ARRAY_LEN(added); i++) {
+        if ((m->added_token_mask & added[i].bit) == 0)
+            continue;
+        uint32_t got = 0;
+        if (ids[i] >= m->vocab_size ||
+            !se_vocab_lookup(model, (const uint8_t *)added[i].text, added[i].len, &got) ||
+            got != ids[i]) {
+            se_error_set(err, SE_ERR_INVALID_FORMAT,
+                         "added token %s is not reachable at its recorded id", added[i].text);
+            rc = SE_ERR_INVALID_FORMAT;
+            goto done;
+        }
     }
 
 done:
@@ -571,6 +671,7 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
     m->subword_prefix_len = read_u32(base, SE_OFF_SUBWORD_PREFIX_LEN);
     m->max_token_chars = read_u32(base, SE_OFF_MAX_TOKEN_CHARS);
     m->max_probe = read_u32(base, SE_OFF_MAX_PROBE);
+    m->added_token_mask = read_u32(base, SE_OFF_ADDED_TOKEN_MASK);
     memcpy(m->subword_prefix, base + SE_OFF_SUBWORD_PREFIX, 8);
 
     if (m->tokenizer_type != SE_TOKENIZER_BERT_WORDPIECE_V1) {
@@ -594,7 +695,7 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
                      m->normalization_type);
         return SE_ERR_INVALID_FORMAT;
     }
-    if (m->truncation_policy != SE_TRUNCATE_IDS_BEFORE_POOLING) {
+    if (m->truncation_policy != SE_TRUNCATE_USABLE_IDS_BEFORE_POOLING) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "truncation policy %u is not supported",
                      m->truncation_policy);
         return SE_ERR_INVALID_FORMAT;
@@ -607,6 +708,11 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown UNK policy %u", m->unk_policy);
         return SE_ERR_INVALID_FORMAT;
     }
+    if ((m->added_token_mask & ~SE_ADDED_TOKEN_MASK_ALL) != 0) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown added-token mask bits 0x%x",
+                     m->added_token_mask & ~SE_ADDED_TOKEN_MASK_ALL);
+        return SE_ERR_INVALID_FORMAT;
+    }
     if (m->empty_policy != SE_EMPTY_ZERO_VECTOR && m->empty_policy != SE_EMPTY_RAISE) {
         se_error_set(err, SE_ERR_INVALID_FORMAT, "unknown empty-input policy %u", m->empty_policy);
         return SE_ERR_INVALID_FORMAT;
@@ -629,8 +735,14 @@ static se_status_t validate(se_model_t *model, se_error_t *err) {
                      m->hash_table_size);
         return SE_ERR_INVALID_FORMAT;
     }
-    if (m->hash_table_size < m->vocab_size) {
-        se_error_set(err, SE_ERR_INVALID_FORMAT, "hash table smaller than vocabulary");
+    if (m->hash_table_size <= m->vocab_size ||
+        (uint64_t)m->vocab_size * 100u > (uint64_t)m->hash_table_size * 70u) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT,
+                     "hash table load factor exceeds the supported 0.70 maximum");
+        return SE_ERR_INVALID_FORMAT;
+    }
+    if (m->max_probe == 0 || m->max_probe > m->hash_table_size) {
+        se_error_set(err, SE_ERR_INVALID_FORMAT, "invalid recorded max probe %u", m->max_probe);
         return SE_ERR_INVALID_FORMAT;
     }
     if (m->subword_prefix_len > 8) {