static_embeddings 0.1.2 → 0.1.4

data/docs/BENCHMARKING.md

@@ -0,0 +1,87 @@
+# Benchmarking
+
+Two different tools, for two different questions.
+
+`benchmark/` answers "how fast is this, and is the difference I am looking at
+real". `samples/` answers "where does the time go" by keeping a process alive
+for a profiler to attach to. Do not use one for the other's job.
+
+```bash
+bundle exec rake benchmark                 # everything
+bundle exec rake benchmark:core_paths
+bundle exec rake benchmark:gvl_threshold
+
+BENCH_MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
+  bundle exec rake benchmark:core_paths
+```
+
+Without `BENCH_MODEL` the demo model is used, which has `dim=8` and a 176-token
+vocabulary. It proves the harness runs; it says nothing about production speed.
+
+## Reading the output
+
+```text
+case                                      ns/op      ops/sec    spread%   alloc/op
+embed words=20                           8540.5       117089       3.0%       1.00
+```
+
+`spread_pct` is the distance between the fastest and slowest repeat. **A
+difference smaller than `spread_pct` is not a result.** On a laptop or a shared
+CI runner it is routinely 5-10%, which is wider than most changes worth making.
+If you want to claim an improvement, either get the spread down or show the
+change across several independent runs.
+
+`alloc/op` is Ruby object allocations per operation. It moves for different
+reasons than time does, and it is the more stable signal of the two.
+
+Generated text inputs are primed with `valid_encoding?` before timing, so these
+benchmarks measure the cached-coderange hot path. Fresh strings from disk or the
+database can pay an extra whole-string UTF-8 scan; see `docs/PERFORMANCE.md`.
+
+Iteration counts are calibrated automatically so every measurement runs for at
+least `BENCH_MIN_SECONDS`. Knobs:
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `BENCH_MODEL` | demo model | model to load |
+| `BENCH_REPEATS` | 7 | repeats per case, median reported |
+| `BENCH_MIN_SECONDS` | 0.25 | minimum duration of one repeat |
+| `BENCH_WARMUP_SECONDS` | 0.05 | warmup before timing |
+| `BENCH_ADAPTIVE` | 1 | set to 0 to use the literal iteration count |
+| `BENCH_FORMAT` | table | set to `kv` for machine-readable lines |
+
+## What the two benchmarks are for
+
+`core_paths` covers tokenize, single embed, batch embed, pooling in isolation
+and top-k in both formats. The `embed` and `embed rotating corpus` rows are
+deliberately both present: the first repeats one string and keeps its matrix
+rows in cache, the second rotates through 2000 texts and does not. The gap
+between them is the cache effect, and quoting the first as throughput is how you
+end up with a number nobody can reproduce.
+
+`gvl_threshold` sweeps input sizes around `SE_GVL_UNLOCK_THRESHOLD`, the point
+where a single `embed` stops running under the GVL and is handed to the batch
+machinery. Look for the step, not the slope. On one x86-64 run the step cost
+about 32% latency and four extra allocations, and it is also the point where a
+second Ruby thread starts making progress at all. Both halves of that trade have
+to be measured on your hardware before the constant is worth changing.
+
+## Traps this repository has already fallen into
+
+**Quoting noise.** An `embed_batch f16` change was reported as a 6% speedup; it
+was inside the run-to-run spread and the next run showed the opposite sign. The
+memory result from the same change was real and reproducible. Report the one you
+can reproduce.
+
+**Cache-resident working sets.** Embedding the same text in a loop keeps a tiny
+slice of the matrix hot. Real corpora do not.
+
+**Logical vs processed bytes.** With truncation active, a 3 MB document is
+tokenized only until `max_tokens` is reached. Any MB/s figure computed from the
+string length divides by bytes that were never read.
+
+**GC state.** `GC.disable` does not stop an in-flight incremental cycle from
+sweeping. The harness leaves GC on and reports allocations instead.
+
+See `docs/PERFORMANCE.md` for the budget itself and where the time actually
+goes.

data/docs/LIMITATIONS.md

@@ -0,0 +1,93 @@
+# Limitations
+
+`static_embeddings` is a runtime for one narrow thing: turning text into a
+vector with a converted Model2Vec / potion model, and scanning a matrix of such
+vectors. It is not a vector database, not a general embedding library, and not a
+tokenizer toolkit.
+
+This page is the answer to "does it do X". If X is here, the answer is no, and
+the entry says whether that is a decision or just unfinished work.
+
+## Models
+
+- **Only `.semb` files from this repository's converter.** Loading a
+  HuggingFace directory at runtime is not supported and will not be. The
+  converter is the only thing that reads third-party files, and it runs offline.
+- **Only the `BERT_WORDPIECE_V1` tokenizer profile.** SentencePiece, BPE, Unigram
+  and byte-level tokenizers are rejected at conversion time rather than
+  approximated.
+- **No model training, distillation or fine-tuning.** Produce the model with
+  upstream `model2vec`, then convert it.
+- **No transformer inference.** Static embeddings have no attention and no
+  context; a word contributes the same row wherever it appears. If you need
+  contextual embeddings, this is the wrong tool.
+
+## Runtime
+
+- **No internal parallelism.** `threads:` is rejected rather than ignored. Split
+  work at the job level and run more application workers.
+- **No Ractor support.** Model, and everything holding native state, are not
+  shareable; `Ractor.make_shareable` raises.
+- **No GPU, no BLAS, no external SIMD library.** The kernels are plain C with
+  compiler autovectorisation, plus a hand-written half-precision path.
+- **No streaming input.** `embed` and `embed_batch` take Ruby `String`s that are
+  already in memory.
+- **`embed_batch` peaks at roughly twice the returned blob**, because the result
+  is built in C memory and then copied into a Ruby `String`. Chunk large
+  corpora.
+- **Big-endian hosts are refused at load.** The header is decoded
+  little-endian explicitly, but the mmapped structures and the float matrix are
+  read in native order.
+
+## Search
+
+- **No index.** `dot_top_k` and `cosine_top_k` are exhaustive scans. There is no
+  ANN, no IVF, no HNSW, no clustering.
+- **No batched queries.** One query per scan; N queries stream the matrix N
+  times.
+- **No candidate filtering.** You cannot restrict a scan to a subset of rows.
+- **No precomputed row norms.** `cosine_top_k` recomputes each row's norm on
+  every query. Normalise your rows once and use `dot_top_k` instead.
+- **No persistence, no ids, no updates.** A matrix is a frozen `String`; mapping
+  rows to documents, appending, and deleting are the caller's problem. If you
+  want that, use pgvector and keep this gem for producing the vectors.
+- **No metric other than dot product and cosine.** No Euclidean, no Manhattan,
+  no Hamming.
+
+## Storage formats
+
+- **`f32` and `f16` only.** No int8, no binary quantisation.
+- **`f16` is a storage encoding, not a different model.** Top-k over an f16
+  matrix can be faster or slower than f32 depending on the decode kernel; see
+  `docs/PERFORMANCE.md`. `embed_batch(format: :f16)` still encodes the blob
+  with the scalar converter.
+- **`f16` rounding is half-up, not ties-to-even.** Blobs written by this gem can
+  differ from NumPy or PyTorch by one ULP on exact halfway values.
+
+## Platforms
+
+- **Windows is built and smoke-tested by cross-compilation under wine, not on a
+  real Windows runner.** The loader uses `CreateFileMapping`, with a
+  read-into-heap fallback.
+- **No JRuby or TruffleRuby.** The gem is a CRuby C extension.
+
+## Verification
+
+- **`load` does not check the SHA-256.** Structural validation always runs, but
+  a bit flip inside the matrix is only caught by `verify: true` or
+  `StaticEmbeddings.verify`.
+- **Parity is per model and per runtime version.** A converted model is not
+  trusted until it has a record in `docs/MODEL_AUDIT.md`, and that record
+  expires when the tokenizer changes.
+
+## Known open questions
+
+These are not decisions, just work that has not been done or measured:
+
+- The `SE_GVL_UNLOCK_THRESHOLD` value has a benchmark now
+  (`rake benchmark:gvl_threshold`) but has not been retuned from it.
+- Top-k keeps its candidates in a sorted array, which degrades for large `k`; a
+  heap would be better somewhere above a few dozen.
+- `madvise(MADV_RANDOM)` and huge pages for the embedding section are untested.
+- The C allocation counters cover the runtime's own allocations only; Ruby heap,
+  fragmentation and mmap residency are outside them.

data/docs/MODEL_AUDIT.md

@@ -11,25 +11,29 @@ runtime half of it even though the file bytes are untouched.
 | runtime | parity | note |
 |---|---|---|
 | 0.1.1 | `parity OK`, recorded below | superseded |
-| 0.1.2 | **not re-run** | required before release |
+| 0.1.2 | **not re-run** | superseded before release |
+| 0.1.3 | `parity OK` | tokenizer / pooling contract; same numbers as 0.1.4 |
+| 0.1.4 | `parity OK`, recorded below | current |
 
 0.1.2 changed `is_control()`, which changes token ids for any input containing
 `U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
 matches HuggingFace, because it is an implementation twin of the C runtime
-written in this repository. Only the oracle can, and the oracle now carries DEL
-and control-character rows it did not have when the result below was recorded —
-so `rows=28` would itself be evidence of a stale run.
+written in this repository. The 0.1.3 release candidate was checked against a
+fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
+characters, Unicode, OOV, long-word and truncation rows. 0.1.4 re-ran that
+oracle against the SIMD L2 runtime on the same snapshot and `.semb`.
 
 Source model:
 
 - Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
-- Python package: `model2vec 0.9.0`
+- Python package: `model2vec 0.9.0` (`tokenizers 0.23.1`, `numpy 2.5.2`)
 - Oracle file: `tmp/model2vec_oracle.json`
-- Oracle rows in this recorded run: `28`
+- Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
-- Runtime at time of this record: `static_embeddings 0.1.1`
+- Runtime at time of this record: `static_embeddings 0.1.4`
 
 Converted `.semb`:
 
@@ -42,16 +46,20 @@ Converted `.semb`:
 Parity command:
 
 ```bash
+python tools/model2vec_oracle.py \
+  ~/.cache/huggingface/hub/models--minishlab--potion-retrieval-32M/snapshots/6fc8051fab2a1e0ee76689cf08c853792ac285e7 \
+  --out tmp/model2vec_oracle.json
+
 bundle exec rake parity \
   MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
   ORACLE=tmp/model2vec_oracle.json
 ```
 
-Parity result:
+Parity result (0.1.4, 2026-08-31):
 
 ```text
-rows=28
-id_rows_checked=28
+rows=31
+id_rows_checked=31
 min_cosine=0.9999999999989528
 max_abs_all=2.980232238769531e-07
 token_id_failures=[]
@@ -59,6 +67,11 @@ vector_failures=[]
 parity OK
 ```
 
+The printed `min_cosine` and `max_abs_all` are identical to the 0.1.3 record.
+Worst vector row is still idx=29 (`"word" + 400 spaces`, 80 800 bytes) at
+`max_abs=2.9802322e-07`. SIMD pairwise double L2 did not move the oracle
+agreement on this corpus.
+
 Decision:
 
 - Tokenization parity: pass
@@ -68,7 +81,8 @@ Decision:
 - Unicode normalization behavior: pass
 - Unknown-word behavior: pass
 - Long input / truncation behavior: pass
+- DEL / control-character behavior: pass
 
-Accepted for runtime and benchmark use **under 0.1.1**. The `.semb` file is
-unchanged and its SHA256 still matches; what expired is the runtime half of the
-record.
+Accepted for runtime and benchmark use **under 0.1.4**. The `.semb` file is
+unchanged and its SHA256 still matches. The runtime half of the record was
+refreshed after the 0.1.4 L2 kernel change.

data/docs/PERFORMANCE.md

@@ -22,33 +22,42 @@ ruby tools/benchmark.rb [model.semb]
 
 ## Where the time actually goes
 
-Tokenization dominates; pooling is a handful of row reads and adds. That is why
-the tokenizer is the file to profile, and why SIMD in the pooling loop is not
-where the wins are — that loop is memory-bound on random row lookups and the
-compiler already vectorises it.
-
-Out-of-vocabulary text is the main cliff. In-vocabulary words hit the
-vocabulary hash directly; everything else falls through to trie-driven subword
-splitting, which on a synthetic all-OOV corpus measured several times the
-per-text cost. Non-English input on an English model pays this on top of the
-`[UNK]` quality problem.
+Which file to profile depends on the model and the text. On a short-dimension
+demo model, or on synthetic all-OOV input, tokenization dominates. On
+`potion-retrieval-32m` (`dim=512`) with in-vocabulary English, pooling is the
+larger share: each token is a random 2 KiB row add, and before 0.1.4 the scalar
+L2 pass showed up as about 19% of an ASCII `embed_batch` sample tree. SIMD in
+that L2 slot was a real win; another unroll of the add loop was not — it is
+memory-bound on those gathers, and the compiler already vectorises it.
+
+Out-of-vocabulary text is the remaining tokenizer cliff. In-vocabulary words
+hit the vocabulary hash directly; everything else falls through to trie-driven
+subword splitting, which on a synthetic all-OOV corpus measured several times
+the per-text cost. Non-English input on an English model pays this on top of
+the `[UNK]` quality problem. Quote `mean_tokens_per_text` and `unk_ratio` from
+the batch samples before comparing texts/s across modes: hashes are slower
+than ASCII on this corpus mostly because they emit ~73 tokens/text against
+~31, not only because they miss the hash.
 
 ## `f16` is a storage trade-off
 
-`format: :f16` halves the bytes a top-k scan streams and doubles the decode
-work. Which one wins is a property of the machine, and this repository's own
-sample runs disagree with each other — same corpus, same `k`, native decode
-kernel in both cases:
+`format: :f16` halves the bytes a top-k scan streams and costs a decode per
+row. Which one wins is a property of the machine and of the decode kernel, and
+this repository's own sample runs disagree with each other — same corpus, same
+`k`, native decode kernel in both cases:
 
 ```text
                      f32        f16
-x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster
-M1 Pro, neon-fp16  2.29 ms    2.88 ms    f16 1.3x slower
+x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster   (F16C kernel, unchanged)
+M1 Pro, neon-fp16  2.36 ms    1.81 ms    f16 1.3x faster   (0.1.4, 16-wide NEON)
 ```
 
-Neither ratio transfers. Confirm `StaticEmbeddings.simd_backend`, then measure
-on the hardware you will run on. On the lookup-table fallback `f16` is usually
-slower than `f32`.
+The M1 Pro row in 0.1.3 was 2.29 / 2.88 ms, f16 1.3x *slower*, with an 8-wide
+decode. Neither ratio transfers. Confirm `StaticEmbeddings.simd_backend`, then
+measure on the hardware you will run on. On the lookup-table fallback `f16` is
+usually slower than `f32`. `embed_batch(format: :f16)` still encodes the
+returned blob with the scalar half converter, so a batch that is not a top-k
+scan can still be slower than `f32` on the same machine.
 
 ## C allocation counters
 
@@ -72,6 +81,14 @@ for those. And the instrumentation is not free: about 13% on `tokenize`, 3% on
 `embed`, within noise on `embed_batch`. A run captured with it is not
 comparable to one captured without it.
 
+## Running the benchmarks
+
+`bundle exec rake benchmark`, or one at a time with `rake benchmark:core_paths`
+and `rake benchmark:gvl_threshold`. Generated text inputs are
+`valid_encoding?`-primed before timing, so these are cached-coderange hot-path
+numbers. Every row carries `spread_pct`, and a difference smaller than that is
+not a result. See `docs/BENCHMARKING.md`.
+
 ## Benchmark hygiene
 
 Three failure modes have bitten this repository already.

data/ext/static_embeddings/se_embed.c

@@ -8,6 +8,7 @@
 #define SE_HAVE_NEON 1
 #elif defined(__SSE__)
 #include <xmmintrin.h>
+#include <emmintrin.h>
 #define SE_HAVE_SSE 1
 #endif
 
@@ -71,15 +72,49 @@ static void scale_copy(float *out, const float *acc, uint32_t dim, float inv) {
         out[j] = acc[j] * inv;
 }
 
-void se_l2_normalize(float *vec, uint32_t dim) {
+static double sum_sq_d(const float *vec, uint32_t dim) {
+    uint32_t j = 0;
     double sum = 0.0;
-    for (uint32_t j = 0; j < dim; j++)
+#if defined(SE_HAVE_NEON) && defined(__aarch64__)
+    float64x2_t s0 = vdupq_n_f64(0.0);
+    float64x2_t s1 = vdupq_n_f64(0.0);
+    for (; j + 7 < dim; j += 8) {
+        float32x4_t a = vld1q_f32(vec + j);
+        float32x4_t b = vld1q_f32(vec + j + 4);
+        float64x2_t a0 = vcvt_f64_f32(vget_low_f32(a));
+        float64x2_t a1 = vcvt_f64_f32(vget_high_f32(a));
+        float64x2_t b0 = vcvt_f64_f32(vget_low_f32(b));
+        float64x2_t b1 = vcvt_f64_f32(vget_high_f32(b));
+        s0 = vaddq_f64(s0, vmulq_f64(a0, a0));
+        s1 = vaddq_f64(s1, vmulq_f64(a1, a1));
+        s0 = vaddq_f64(s0, vmulq_f64(b0, b0));
+        s1 = vaddq_f64(s1, vmulq_f64(b1, b1));
+    }
+    sum = vaddvq_f64(vaddq_f64(s0, s1));
+#elif defined(SE_HAVE_SSE)
+    __m128d s0 = _mm_setzero_pd();
+    __m128d s1 = _mm_setzero_pd();
+    for (; j + 3 < dim; j += 4) {
+        __m128 v = _mm_loadu_ps(vec + j);
+        __m128d lo = _mm_cvtps_pd(v);
+        __m128d hi = _mm_cvtps_pd(_mm_movehl_ps(v, v));
+        s0 = _mm_add_pd(s0, _mm_mul_pd(lo, lo));
+        s1 = _mm_add_pd(s1, _mm_mul_pd(hi, hi));
+    }
+    double tmp[2];
+    _mm_storeu_pd(tmp, _mm_add_pd(s0, s1));
+    sum = tmp[0] + tmp[1];
+#endif
+    for (; j < dim; j++)
         sum += (double)vec[j] * (double)vec[j];
+    return sum;
+}
+
+void se_l2_normalize(float *vec, uint32_t dim) {
+    double sum = sum_sq_d(vec, dim);
     if (sum <= 0.0)
         return;
-    float inv = (float)(1.0 / sqrt(sum));
-    for (uint32_t j = 0; j < dim; j++)
-        vec[j] *= inv;
+    scale_copy(vec, vec, dim, (float)(1.0 / sqrt(sum)));
 }
 
 static se_status_t embed_ids_core(const se_model_t *model, se_scratch_t *sc, const uint32_t *ids,
@@ -128,8 +163,12 @@ static se_status_t embed_ids_core(const se_model_t *model, se_scratch_t *sc, con
     }
 
     if (model->meta.normalization_type == SE_NORMALIZATION_L2) {
-        scale_copy(out, acc, dim, 1.0f);
-        se_l2_normalize(out, dim);
+        double sum = sum_sq_d(acc, dim);
+        if (sum <= 0.0) {
+            memset(out, 0, (size_t)dim * sizeof(float));
+            return SE_OK;
+        }
+        scale_copy(out, acc, dim, (float)(1.0 / sqrt(sum)));
     } else {
         const float inv = 1.0f / (float)used;
         scale_copy(out, acc, dim, inv);

data/ext/static_embeddings/se_f16.c

@@ -221,17 +221,31 @@ static float dot_product_f16_neon(const float *q, const uint8_t *row, size_t dim
     size_t j = 0;
     float32x4_t a0 = vdupq_n_f32(0.0f);
     float32x4_t a1 = vdupq_n_f32(0.0f);
+    float32x4_t a2 = vdupq_n_f32(0.0f);
+    float32x4_t a3 = vdupq_n_f32(0.0f);
+    for (; j + 15 < dim; j += 16) {
+        float16x8_t h0 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + j * 2)));
+        float16x8_t h1 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + (j + 8) * 2)));
+        float32x4_t r0 = vcvt_f32_f16(vget_low_f16(h0));
+        float32x4_t r1 = vcvt_f32_f16(vget_high_f16(h0));
+        float32x4_t r2 = vcvt_f32_f16(vget_low_f16(h1));
+        float32x4_t r3 = vcvt_f32_f16(vget_high_f16(h1));
+        a0 = vmlaq_f32(a0, vld1q_f32(q + j), r0);
+        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), r1);
+        a2 = vmlaq_f32(a2, vld1q_f32(q + j + 8), r2);
+        a3 = vmlaq_f32(a3, vld1q_f32(q + j + 12), r3);
+    }
     for (; j + 7 < dim; j += 8) {
         float16x4_t h0 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + j * 2)));
         float16x4_t h1 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + (j + 4) * 2)));
-        float32x4_t r0 = vcvt_f32_f16(h0);
-        float32x4_t r1 = vcvt_f32_f16(h1);
-        a0 = vmlaq_f32(a0, vld1q_f32(q + j), r0);
-        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), r1);
+        a0 = vmlaq_f32(a0, vld1q_f32(q + j), vcvt_f32_f16(h0));
+        a1 = vmlaq_f32(a1, vld1q_f32(q + j + 4), vcvt_f32_f16(h1));
     }
-    float32x4_t sumv = vaddq_f32(a0, a1);
+    float32x4_t sumv = vaddq_f32(vaddq_f32(a0, a1), vaddq_f32(a2, a3));
     float dot = vaddvq_f32(sumv);
     for (; j < dim; j++)
         dot += q[j] * se_f16_bits_to_float(read_u16le(row + j * 2));
@@ -243,8 +257,30 @@ static float dot_and_row_sq_f16_neon(const float *q, const uint8_t *row, size_t
     size_t j = 0;
     float32x4_t d0 = vdupq_n_f32(0.0f);
     float32x4_t d1 = vdupq_n_f32(0.0f);
+    float32x4_t d2 = vdupq_n_f32(0.0f);
+    float32x4_t d3 = vdupq_n_f32(0.0f);
     float32x4_t s0 = vdupq_n_f32(0.0f);
     float32x4_t s1 = vdupq_n_f32(0.0f);
+    float32x4_t s2 = vdupq_n_f32(0.0f);
+    float32x4_t s3 = vdupq_n_f32(0.0f);
+    for (; j + 15 < dim; j += 16) {
+        float16x8_t h0 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + j * 2)));
+        float16x8_t h1 =
+            vreinterpretq_f16_u16(vld1q_u16((const uint16_t *)(const void *)(row + (j + 8) * 2)));
+        float32x4_t r0 = vcvt_f32_f16(vget_low_f16(h0));
+        float32x4_t r1 = vcvt_f32_f16(vget_high_f16(h0));
+        float32x4_t r2 = vcvt_f32_f16(vget_low_f16(h1));
+        float32x4_t r3 = vcvt_f32_f16(vget_high_f16(h1));
+        d0 = vmlaq_f32(d0, vld1q_f32(q + j), r0);
+        d1 = vmlaq_f32(d1, vld1q_f32(q + j + 4), r1);
+        d2 = vmlaq_f32(d2, vld1q_f32(q + j + 8), r2);
+        d3 = vmlaq_f32(d3, vld1q_f32(q + j + 12), r3);
+        s0 = vmlaq_f32(s0, r0, r0);
+        s1 = vmlaq_f32(s1, r1, r1);
+        s2 = vmlaq_f32(s2, r2, r2);
+        s3 = vmlaq_f32(s3, r3, r3);
+    }
     for (; j + 7 < dim; j += 8) {
         float16x4_t h0 =
             vreinterpret_f16_u16(vld1_u16((const uint16_t *)(const void *)(row + j * 2)));
@@ -257,8 +293,8 @@ static float dot_and_row_sq_f16_neon(const float *q, const uint8_t *row, size_t
         s0 = vmlaq_f32(s0, r0, r0);
         s1 = vmlaq_f32(s1, r1, r1);
     }
-    float dot = vaddvq_f32(vaddq_f32(d0, d1));
-    float row_sq = vaddvq_f32(vaddq_f32(s0, s1));
+    float dot = vaddvq_f32(vaddq_f32(vaddq_f32(d0, d1), vaddq_f32(d2, d3)));
+    float row_sq = vaddvq_f32(vaddq_f32(vaddq_f32(s0, s1), vaddq_f32(s2, s3)));
     for (; j < dim; j++) {
         float r = se_f16_bits_to_float(read_u16le(row + j * 2));
         dot += q[j] * r;

data/ext/static_embeddings/se_format.c

@@ -61,6 +61,34 @@ static uint32_t hash_bytes_continue(uint32_t h, const uint8_t *data, size_t len)
     return h;
 }
 
+static int se_memeq(const uint8_t *a, const uint8_t *b, size_t n) {
+    while (n >= 8) {
+        uint64_t ua, ub;
+        memcpy(&ua, a, 8);
+        memcpy(&ub, b, 8);
+        if (ua != ub)
+            return 0;
+        a += 8;
+        b += 8;
+        n -= 8;
+    }
+    if (n >= 4) {
+        uint32_t ua, ub;
+        memcpy(&ua, a, 4);
+        memcpy(&ub, b, 4);
+        if (ua != ub)
+            return 0;
+        a += 4;
+        b += 4;
+        n -= 4;
+    }
+    while (n--) {
+        if (*a++ != *b++)
+            return 0;
+    }
+    return 1;
+}
+
 int se_vocab_lookup_piece(const se_model_t *model, const uint8_t *prefix, size_t prefix_len,
                           const uint8_t *bytes, size_t len, uint32_t *id_out) {
     if (prefix_len > UINT32_MAX || len > UINT32_MAX || prefix_len > UINT32_MAX - len)
@@ -79,9 +107,9 @@ int se_vocab_lookup_piece(const se_model_t *model, const uint8_t *prefix, size_t
         if (slot->token_id == SE_SLOT_EMPTY)
             return 0;
         if (slot->hash == h && slot->str_len == total_len) {
-            const char *token = model->vocab_strings + slot->str_off;
-            if ((prefix_len == 0 || memcmp(token, prefix, prefix_len) == 0) &&
-                (len == 0 || memcmp(token + prefix_len, bytes, len) == 0)) {
+            const uint8_t *token = (const uint8_t *)(model->vocab_strings + slot->str_off);
+            if ((prefix_len == 0 || se_memeq(token, prefix, prefix_len)) &&
+                (len == 0 || se_memeq(token + prefix_len, bytes, len))) {
                 *id_out = slot->token_id;
                 return 1;
             }
@@ -103,6 +131,17 @@ static inline uint32_t trie_find_child(const se_trie_t *trie, const se_trie_node
     if (count == 0)
         return SE_SLOT_EMPTY;
 
+    if (count == 1)
+        return trie->edges[start].byte == byte ? trie->edges[start].child : SE_SLOT_EMPTY;
+
+    if (count == 2) {
+        if (trie->edges[start].byte == byte)
+            return trie->edges[start].child;
+        if (trie->edges[start + 1u].byte == byte)
+            return trie->edges[start + 1u].child;
+        return SE_SLOT_EMPTY;
+    }
+
     if (count <= 8) {
         for (uint32_t i = 0; i < count; i++) {
             const se_trie_edge_t *edge = &trie->edges[start + i];
@@ -342,6 +381,37 @@ static se_status_t validate_norm_ranges(const se_range_t *ranges, uint32_t count
     return SE_OK;
 }
 
+static void fill_map256(const se_map_entry_t **out, const se_map_entry_t *entries, uint32_t count) {
+    memset(out, 0, 256 * sizeof(*out));
+    for (uint32_t i = 0; i < count; i++) {
+        if (entries[i].cp < 256)
+            out[entries[i].cp] = &entries[i];
+    }
+}
+
+static void fill_range256(uint8_t *out, const se_range_t *ranges, uint32_t count) {
+    memset(out, 0, 256);
+    for (uint32_t i = 0; i < count; i++) {
+        uint32_t lo = ranges[i].lo;
+        uint32_t hi = ranges[i].hi;
+        if (lo > 255)
+            continue;
+        if (hi > 255)
+            hi = 255;
+        for (uint32_t cp = lo; cp <= hi; cp++)
+            out[cp] = 1;
+    }
+}
+
+static void prepare_norm_fast_tables(se_model_t *model) {
+    fill_map256(model->norm.lower256, model->norm.lower, model->norm.lower_count);
+    fill_map256(model->norm.nfd256, model->norm.nfd, model->norm.nfd_count);
+    fill_range256(model->norm.mn256, model->norm.mn, model->norm.mn_count);
+    fill_range256(model->norm.punct256, model->norm.punct, model->norm.punct_count);
+    fill_range256(model->norm.control256, model->norm.control, model->norm.control_count);
+    fill_range256(model->norm.whitespace256, model->norm.whitespace, model->norm.whitespace_count);
+}
+
 static se_status_t parse_norm_tables(se_model_t *model, const uint8_t *base, struct section sec,
                                      se_error_t *err) {
     memset(&model->norm, 0, sizeof(model->norm));
@@ -417,8 +487,13 @@ static se_status_t parse_norm_tables(se_model_t *model, const uint8_t *base, str
     rc = validate_norm_ranges(model->norm.control, model->norm.control_count, "control", err);
     if (rc != SE_OK)
         return rc;
-    return validate_norm_ranges(model->norm.whitespace, model->norm.whitespace_count, "whitespace",
-                                err);
+    rc = validate_norm_ranges(model->norm.whitespace, model->norm.whitespace_count, "whitespace",
+                              err);
+    if (rc != SE_OK)
+        return rc;
+
+    prepare_norm_fast_tables(model);
+    return SE_OK;
 }
 
 static se_status_t validate_vocab_hash(const se_model_t *model, struct section vocab_strings,
@@ -845,5 +920,8 @@ size_t se_model_memsize(const se_model_t *model) {
     if (!model)
         return 0;
 
+    if (model->mapped)
+        return sizeof(se_model_t);
+
     return sizeof(se_model_t) + model->map_size;
 }

data/ext/static_embeddings/se_internal.h

@@ -136,6 +136,12 @@ typedef struct {
     uint32_t control_count;
     const se_range_t *whitespace;
     uint32_t whitespace_count;
+    const se_map_entry_t *lower256[256];
+    const se_map_entry_t *nfd256[256];
+    uint8_t mn256[256];
+    uint8_t punct256[256];
+    uint8_t control256[256];
+    uint8_t whitespace256[256];
 } se_norm_tables_t;
 
 typedef struct {
@@ -382,6 +388,9 @@ static inline int se_is_ascii_boundary(uint32_t cp) {
 void se_scratch_init(se_scratch_t *s);
 void se_scratch_free(se_scratch_t *s);
 int se_scratch_reserve(se_scratch_t *s, uint32_t dim);
+se_scratch_t *se_scratch_acquire(uint32_t dim);
+void se_scratch_release(se_scratch_t *s);
+void se_scratch_drop_thread(void);
 
 size_t se_prefix_boundary_len(const se_model_t *model, const uint8_t *input, size_t input_len,
                               size_t target, size_t backscan);