static_embeddings 0.1.1

data/docs/ARCHITECTURE.md

@@ -0,0 +1,278 @@
+# Architecture
+
+## The shape of the thing
+
+```
+HuggingFace / Model2Vec files          offline, once, on your machine
+  tokenizer.json, config.json,
+  model.safetensors
+        |
+        v
+  Converter (pure Ruby, strict)   <--- all parsing, all validation, all
+        |                              Unicode table generation happens here
+        v
+  model.semb                       <--- flat, versioned, mmap-able
+        |
+        v
+  C runtime                        <--- mmap + bounds checks + tokenize +
+  (no JSON, no safetensors,             lookup + pool. Nothing else.
+   no Unicode database, no net)
+```
+
+The single most important decision: **the runtime reads only our own format.**
+Everything expensive, fragile or security-sensitive about reading third-party
+model files happens once, offline, in a language where it is easy to get
+right. What remains in C is a bounds-checked mmap and three loops.
+
+## `.semb` v2
+
+Little-endian throughout. 320-byte header, then sections aligned to 64 bytes.
+
+| Offset | Size | Field |
+|---|---|---|
+| 0 | 8 | magic `SEMBv1\0\0` |
+| 8 | 4 | format_version |
+| 12 | 4 | header_size (320) |
+| 16 | 4 | flags |
+| 20 | 4 | dim |
+| 24 | 4 | vocab_size |
+| 28 | 4 | tokenizer_type (1 = BERT_WORDPIECE_V1) |
+| 32 | 4 | embedding_dtype (1 = F32) |
+| 36 | 4 | pooling_type (1 = MEAN) |
+| 40 | 4 | normalization_type (0 none, 1 = L2) |
+| 44 | 4 | **max_tokens_default** |
+| 48 | 4 | truncation_policy |
+| 52 | 4 | add_special_tokens |
+| 56 | 4 | unk_policy (0 include, 1 drop) |
+| 60 | 4 | empty_policy (0 zero vector, 1 raise) |
+| 64 | 4 | do_lower_case |
+| 68 | 4 | strip_accents |
+| 72 | 4 | handle_chinese_chars |
+| 76 | 4 | clean_text |
+| 80 | 4 | max_input_chars_per_word |
+| 84–100 | 20 | pad/unk/cls/sep/mask ids |
+| 104 | 4 | hash_table_size (power of two) |
+| 108 | 4 | hash_seed |
+| 112 | 4 | subword_prefix_len |
+| 116 | 8 | subword_prefix bytes |
+| 124 | 4 | max_token_chars |
+| 128–207 | 80 | five (offset, size) u64 pairs: `vocab_strings`, `vocab_hash`, `embeddings`, `norm_tables`, `provenance` |
+| 208–239 | 32 | two (offset, size) u64 pairs: `root_trie`, `continuation_trie` |
+| 240 | 32 | sha256 of the file with these 32 bytes zeroed |
+| 304 | 4 | max_probe observed in the vocabulary hash |
+
+Sections, in fixed order: `vocab_strings`, `vocab_hash`, `embeddings`,
+`norm_tables`, `provenance`, `root_trie`, `continuation_trie`.
+
+Extension points are the enums and `format_version`, not stub code. `dtype`
+already has a slot for I8; there is no dead I8 path in C.
+
+### Why the semantics live in the header
+
+`max_tokens_default`, `unk_policy`, `empty_policy` and the normalizer flags
+are properties of the *model*, not options the caller guesses. The reference
+implementation truncates at 512 tokens; a model card's `model_max_length` of
+1,000,000 is a different number entirely, and confusing them silently changes
+every vector for long chunks. Anything that can silently change output is a
+recorded field.
+
+## Vocabulary lookup and WordPiece trie
+
+The converter still writes a finished open-addressing hash table into the
+file: 16-byte slots of `{hash, str_off, str_len, token_id}`, power-of-two size,
+load factor ≤ 0.70, linear probing, FNV-1a with a fixed seed. The hash table is
+kept for validation, direct lookup and debugging.
+
+The hot WordPiece path no longer performs repeated hash lookups for every
+`end--` candidate. Format v2 also stores two converter-built, mmap-readable
+tries inspired by double-array trie libraries such as libdatrie:
+
+- `root_trie` for tokens that can start a word;
+- `continuation_trie` for `##token` entries stored without the `##` prefix.
+
+At runtime `append_wordpiece` walks the relevant trie once, records the longest
+terminal node and emits that token id. This matches the shape of WordPiece much
+better than exact hash lookup: longest-prefix matching is one forward pass
+rather than many candidate hashes and memcmps.
+
+Loading a model still performs **zero runtime insertions**. It is `mmap` plus a
+validation pass over the hash table, trie node ranges and trie edge ordering.
+The vocabulary structures live in the page cache, shared between every forked
+Puma worker.
+
+Rejected alternatives:
+
+- **khash / any runtime hash map.** Would rebuild 250k entries at every boot
+  and allocate outside the mapping — exactly what the offline format exists
+  to avoid.
+- **Sorted table + binary search.** ~18 cache misses per probe at 250k
+  entries versus one or two for open addressing.
+- **Vendored trie runtime.** MARISA, cedar and libdatrie are useful references,
+  but the runtime needs a small `.semb` section that can be bounds-checked and
+  mmap-read directly, not an external object format or system dependency.
+
+## Unicode without a Unicode library
+
+`BertNormalizer` needs NFD, simple lowercase, and the Mn / P / C / Zs
+categories. Rather than vendoring utf8proc, the converter generates exactly
+those tables from the Ruby VM's own Unicode data and writes them into the
+model file; the runtime binary-searches them.
+
+This removes the last third-party dependency, and it sidesteps a specific
+trap: utf8proc's case *folding* is not the same as case *lowering*. Folding
+maps `ß` to `ss`; `str::to_lowercase`, which HF uses, does not. A gem that
+picked the wrong one would produce plausible, slightly wrong vectors forever.
+
+The Ruby build that generated the tables is stamped into provenance, so a
+model file and its normalizer can never drift apart unnoticed.
+
+## The embedding kernel
+
+```
+tokenize -> ids
+truncate to max_tokens          (before pooling — the reference contract)
+drop [UNK] if unk_policy = DROP
+acc[j] += row[j] for each id, in token order
+divide by the number of pooled rows
+L2 normalize if the model says so
+```
+
+Deliberately absent:
+
+- **Deduplicating repeated token ids** and multiplying by a count. It saves
+  memory traffic and changes the float rounding, in the one place where we are
+  trying to stay inside 1e-6 of a reference. Not worth it.
+
+The runtime computes embeddings in float32. Returned storage is selected by the
+caller's `format:` option: `:f32` returns the native 4-byte components and `:f16`
+encodes the same vector as little-endian IEEE float16 components. This is a
+storage/transport choice, not a different model. `embed_array` and
+`embed_batch_arrays` decode whichever storage format was requested back to Ruby
+`Float` values.
+
+The f32 compute path uses small explicit SIMD kernels where they keep the format
+simple: row accumulation and scaling use NEON/SSE when the compiler target
+exposes them, with scalar fallback everywhere else. `dot_top_k`/`cosine_top_k`
+share the SIMD dot kernel for `format: :f32`; `cosine_top_k` additionally
+accumulates each row's sum of squares so it can divide by the true norms.
+`format: :f16` top-k decodes half components on the fly. What is still
+deliberately absent is a dependency on BLAS: pooling is gathering random
+embedding rows, not a dense matrix multiply.
+
+A matrix at or above `SE_TOPK_GVL_UNLOCK_THRESHOLD` (1 MiB) is scanned with the
+GVL released and must therefore be frozen. The alternatives were both bad:
+copying a corpus-sized blob on every query defeats the purpose of the API, and
+`rb_str_locktmp` is an exclusive lock, so it turns a read-only shared corpus
+into something exactly one thread at a time may search. Freezing removes the
+mutation hazard outright and keeps the scan lock-free. Because a raw blob has no
+dimension or format tag, `dim:` is required and the query length is checked
+against it; that is what stops an f32 query and an f16 matrix from dividing out
+to a consistent-looking row count.
+
+Top-k insertion uses `!(score > worst)` rather than `score <= worst` so that a
+`NaN` score is skipped. With the naive comparison a single `NaN` row would fall
+through the early exit, settle at the last slot and then make every subsequent
+comparison false, turning an O(1) reject into an O(k) insert for the rest of the
+scan and leaking `NaN` into the result.
+
+## Large inputs and the prefix window
+
+Ruby strings must be copied into C-owned memory before the GVL is released, and
+that copy runs with the GVL held. Copying a whole multi-megabyte document
+therefore serialised the process on work that the tokenizer would discard after
+`max_tokens` anyway.
+
+`embed` and `embed_batch` instead copy a leading slice. The budget starts at
+`max_tokens * 16` bytes clamped to `[4096, 65536]`, and the slice is cut at a
+position where the tokenizer would have started a fresh word. A result is
+accepted only when that slice actually reached `max_tokens`; otherwise the
+budget doubles and the text is tokenized again. Growth is geometric, so the
+pathological case costs about twice the single-pass work.
+
+Three properties make this exact rather than approximate:
+
+- The scan only stops on UTF-8 lead bytes, so the cut always lands on a
+  codepoint boundary.
+- WordPiece segmentation is word-local, so the token stream of a prefix cut at a
+  word boundary is a prefix of the token stream of the whole document.
+- Normalization carries no state across a word boundary: combining marks are
+  dropped independently, CJK characters are wrapped in spaces individually, and
+  control characters are deleted pointwise.
+
+The boundary predicate is `se_prefix_boundary_len`, and it lives in
+`se_tokenizer.c` next to the tokenizer so it can call the very same
+`is_whitespace` / `is_punct` / `se_is_cjk` predicates rather than a parallel
+copy of them. A position is a legal cut when the codepoint ending there is
+whitespace or punctuation, or when either side of it is a CJK codepoint - CJK is
+space-wrapped by `emit_cleaned`, so both of its edges are word boundaries.
+Codepoints that the normaliser rewrites (lowercase map, NFD map, dropped
+combining marks) are skipped: their class after normalisation is not necessarily
+their class before it.
+
+Restricting the cut to ASCII, as 0.1.0 did, was correct but degenerate: a CJK or
+NBSP-separated document has no ASCII byte anywhere, so the budget grew to the
+full length and the whole input was copied under the GVL after all. That is why
+the predicate is Unicode-aware. Text with genuinely no legal cut inside the scan
+window - one enormous word, a run of combining marks - still falls back to a
+full copy, because there is nothing else that would be correct.
+
+`test/prefix_chunking_test.rb` sweeps every printable ASCII separator, checks
+CJK, NBSP and Unicode-punctuation documents against
+`embed_token_ids(tokenize(text))`, and asserts that quadrupling the input does
+not triple the cost.
+
+With `max_tokens: false` there is nothing to truncate and the whole input is
+copied.
+
+## Concurrency
+
+Small calls run inline. Large `embed_batch` calls copy Ruby input into C-owned
+memory and then run outside the GVL. This is enough for Puma's threaded model:
+one request can compute embeddings while other Ruby threads keep running.
+
+When a fiber scheduler is installed, large batches are handed to one Ruby
+thread before the caller waits, so the reactor is not pinned by the C loop. This
+costs one OS thread per call, so under high concurrency batching beats many
+small `embed` calls. The runtime intentionally does not expose internal
+`threads:` parallelism; offline indexing should split work at the
+job/application layer.
+
+Cancellation is cooperative: the tokenizer checks a flag every 1024 codepoints
+(counted by iteration, not by byte offset, so the interval does not depend on
+how wide the input's codepoints are), the pooling loop checks every 256 rows,
+and the top-k scan every 1024 rows. `unblock_cancel` sets that flag when Ruby
+interrupts a GVL-free region.
+
+No global mutable state exists in C. The model is immutable after load and
+scratch buffers are per call.
+
+## Where this fits in a RAG pipeline
+
+Dense retrieval with static embeddings is a weaker first stage than a
+transformer. The cheap fix is fusion, not a bigger model: BM25 and dense
+vectors fail in different ways — BM25 misses paraphrases, dense misses rare
+identifiers and exact phrases — and Reciprocal Rank Fusion over both recovers
+most of the difference at zero additional cost, since SQLite's FTS5 gives you
+BM25 for free.
+
+```
+chunks -> FTS5 (BM25) ---\
+                          >-- RRF --> top 20 -> optional cross-encoder rerank
+chunks -> .semb vectors --/
+```
+
+## Open questions before v1.0
+
+1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The 32M family
+   has a larger vocabulary than the bge-base tokenizer it was distilled from,
+   which means tokens were added somewhere. If they live in `added_tokens`
+   rather than `model.vocab`, HF matches them with a separate trie pass over
+   raw text that this runtime does not implement. The converter refuses such
+   models today; whether it has to is an audit question.
+2. **Russian.** Distilling a multilingual teacher into a WordPiece vocabulary
+   we control keeps the pure-C path, but skips the Tokenlearn pre-training
+   that gives the published potion models their quality. That gap has to be
+   measured before the multilingual story is real.
+3. **int8.** The format has the slot; the runtime does not have the path.
+   f32 stays the reference forever regardless; f16 is only a returned storage
+   encoding.

data/docs/MODEL_AUDIT.md

@@ -0,0 +1,61 @@
+# Model Audit
+
+This document records the trust and parity status of converted `.semb` models.
+A converted production model is accepted only after parity against the upstream
+Python `model2vec.StaticModel` implementation is recorded here.
+
+## potion-retrieval-32m
+
+Status: audited / accepted
+
+Source model:
+
+- Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Oracle implementation: `model2vec.StaticModel.from_pretrained`
+- Python package: `model2vec 0.9.0`
+- Oracle file: `tmp/model2vec_oracle.json`
+- Oracle rows: `28`
+- Oracle dimension: `512`
+- Oracle max length: `512`
+- Runtime checked with: `static_embeddings 0.1.1`
+- Runtime source note: re-run parity after changing tokenizer, normalizer, prefix-window, pooling, or output-normalization code.
+
+Converted `.semb`:
+
+- Path: `$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb`
+- Format version: `2`
+- Header size: `320`
+- Bytes: `135411608`
+- SHA256: `79e087863d2bab825779fd7de3574e5625542ef6a5ecad33fe681ea16d4b3ab0`
+
+Parity command:
+
+```bash
+bundle exec rake parity \
+  MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
+  ORACLE=tmp/model2vec_oracle.json
+```
+
+Parity result:
+
+```text
+rows=28
+id_rows_checked=28
+min_cosine=0.9999999999989528
+max_abs_all=2.980232238769531e-07
+token_id_failures=[]
+vector_failures=[]
+parity OK
+```
+
+Decision:
+
+- Tokenization parity: pass
+- Vector parity: pass
+- Empty input behavior: pass
+- Whitespace behavior: pass
+- Unicode normalization behavior: pass
+- Unknown-word behavior: pass
+- Long input / truncation behavior: pass
+
+This converted model is accepted for runtime and benchmark use.

data/docs/PERFORMANCE.md

@@ -0,0 +1,60 @@
+# Performance budget
+
+Absolute latency numbers are not comparable across models: `potion-base-2M`
+and `potion-retrieval-32M` differ by roughly 8x in row width, so swapping the
+model would read as a regression. The budget is therefore normalised.
+
+Run `ruby tools/benchmark.rb [model.semb]`.
+
+## Metrics tracked
+
+| Metric | Unit | Why |
+|---|---|---|
+| tokenization | ns / input byte | independent of `dim`; catches normalizer regressions |
+| pooling | ns / token / 100 dims | comparable across models |
+| single embed | µs / call | the latency a query pays |
+| batch throughput | texts/s, tokens/s | indexing capacity |
+| RSS overhead | bytes beyond the mapped file | must stay O(scratch), not O(model) |
+| cold vs warm | first query after a cold page cache | mmap means the first pass faults in the matrix |
+| time to first query | ms from `load` to first vector | what a Puma `before_fork` or a serverless cold start pays |
+
+`samples/cold_start.rb` measures the last row. It is single shot on purpose:
+`load` walks the whole vocabulary hash table and both tries exactly once, and
+`warmup!` only faults pages in once. Drop the page cache first
+(`sudo purge` on macOS, `echo 3 > /proc/sys/vm/drop_caches` on Linux) or the
+output will label itself a warm start.
+
+Every other sample under `samples/` runs a hot loop and measures steady state.
+`warmup_hot_path.rb` in particular re-touches pages that are already resident;
+its `mapped_range_gb_per_sec` is address-range coverage, not memory bandwidth,
+because warmup reads one byte per page.
+
+## CI status
+
+The CI benchmark job is informational. Treat regression gates as meaningful only when a checked-in baseline is compared on the same runner class. Absolute thresholds on shared CI hardware produce noise, not signal.
+
+## Benchmark hygiene
+
+Two failure modes have bitten this repository already and are worth stating:
+
+- **Logical vs processed bytes.** With truncation active, a 3 MB document is
+  tokenized only until `max_tokens` is reached. Any "MB/s of input" figure
+  divides by bytes that were never read. Samples that still print one label it
+  `logical_input_mb_per_sec` and set `truncation_active`.
+- **Resident working sets.** Embedding the same text, or the same frozen id
+  array, in a loop keeps a tiny slice of the matrix in cache and overstates
+  throughput. `random_pooling_hot_path.rb` rotates through many id sets and
+  prints `touched_matrix_mb` so the reader can check it exceeds the last level
+  cache.
+
+`GC.disable` is off by default in the sample harness: it does not stop an
+already started incremental cycle from sweeping, so zeroed GC counters used to
+appear next to profiles full of `gc_sweep`. Set `GC_DISABLE=1` deliberately and
+read `gc_disabled=` before quoting `gc_delta`.
+
+## Where the time actually goes
+
+For static embeddings, tokenization dominates: pooling is a handful of row
+reads and adds. That is why the tokenizer is the file to profile, and why
+SIMD in the pooling loop is not where the wins are — the loop is memory-bound
+on random row lookups, and the compiler already vectorises it.

data/exe/static_embeddings

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require "static_embeddings"
+require "static_embeddings/cli"
+
+exit(StaticEmbeddings::CLI.run(ARGV) || 0)

data/ext/static_embeddings/extconf.rb

@@ -0,0 +1,44 @@
+require "mkmf"
+
+def drop_unsupported_ruby_warnflags
+  return if RbConfig::CONFIG["CC"].to_s.include?("clang")
+
+  flags = %w[
+    -Wno-self-assign
+    -Wno-parentheses-equality
+    -Wno-constant-logical-operand
+  ]
+
+  [RbConfig::CONFIG, RbConfig::MAKEFILE_CONFIG].each do |config|
+    warnflags = config["warnflags"].to_s.split
+    config["warnflags"] = (warnflags - flags).join(" ")
+  end
+end
+
+drop_unsupported_ruby_warnflags
+
+MINIMUM_RUBY_VERSION = Gem::Version.new("3.1.0")
+if Gem::Version.new(RUBY_VERSION) < MINIMUM_RUBY_VERSION
+  abort "static_embeddings requires Ruby >= #{MINIMUM_RUBY_VERSION}; current Ruby is #{RUBY_VERSION}"
+end
+
+$srcs = %w[
+  static_embeddings.c
+  se_format.c
+  se_unicode.c
+  se_tokenizer.c
+  se_embed.c
+]
+
+have_header("ruby/fiber/scheduler.h")
+have_header("sys/mman.h")
+have_func("madvise", "sys/mman.h")
+
+unless [RbConfig::CONFIG["LIBS"], RbConfig::CONFIG["LIBRUBYARG_SHARED"], RbConfig::CONFIG["DLDFLAGS"]].compact.any? { |v| v.include?("-lm") }
+  have_library("m")
+end
+
+$CFLAGS += " -O3 -std=gnu99 -fvisibility=hidden"
+$CFLAGS += " -Wall -Wextra -Wno-unused-parameter"
+
+create_makefile("static_embeddings/static_embeddings")

data/ext/static_embeddings/se_embed.c

@@ -0,0 +1,192 @@
+#include "se_internal.h"
+
+#include <math.h>
+#include <string.h>
+
+#if defined(__ARM_NEON) || defined(__ARM_NEON__)
+#include <arm_neon.h>
+#define SE_HAVE_NEON 1
+#elif defined(__SSE__)
+#include <xmmintrin.h>
+#define SE_HAVE_SSE 1
+#endif
+
+#if defined(HAVE_MADVISE) && !defined(_WIN32)
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define SE_PREFETCH(addr) __builtin_prefetch((addr), 0, 1)
+#else
+#define SE_PREFETCH(addr)
+#endif
+
+#define SE_PREFETCH_DISTANCE 4
+
+static void add_row(float *acc, const float *row, uint32_t dim) {
+    uint32_t j = 0;
+#if defined(SE_HAVE_NEON)
+    for (; j + 15 < dim; j += 16) {
+        vst1q_f32(acc + j, vaddq_f32(vld1q_f32(acc + j), vld1q_f32(row + j)));
+        vst1q_f32(acc + j + 4, vaddq_f32(vld1q_f32(acc + j + 4), vld1q_f32(row + j + 4)));
+        vst1q_f32(acc + j + 8, vaddq_f32(vld1q_f32(acc + j + 8), vld1q_f32(row + j + 8)));
+        vst1q_f32(acc + j + 12, vaddq_f32(vld1q_f32(acc + j + 12), vld1q_f32(row + j + 12)));
+    }
+#elif defined(SE_HAVE_SSE)
+    for (; j + 15 < dim; j += 16) {
+        _mm_storeu_ps(acc + j, _mm_add_ps(_mm_loadu_ps(acc + j), _mm_loadu_ps(row + j)));
+        _mm_storeu_ps(acc + j + 4,
+                      _mm_add_ps(_mm_loadu_ps(acc + j + 4), _mm_loadu_ps(row + j + 4)));
+        _mm_storeu_ps(acc + j + 8,
+                      _mm_add_ps(_mm_loadu_ps(acc + j + 8), _mm_loadu_ps(row + j + 8)));
+        _mm_storeu_ps(acc + j + 12,
+                      _mm_add_ps(_mm_loadu_ps(acc + j + 12), _mm_loadu_ps(row + j + 12)));
+    }
+#endif
+    for (; j < dim; j++)
+        acc[j] += row[j];
+}
+
+static void scale_copy(float *out, const float *acc, uint32_t dim, float inv) {
+    uint32_t j = 0;
+#if defined(SE_HAVE_NEON)
+    float32x4_t vinv = vdupq_n_f32(inv);
+    for (; j + 15 < dim; j += 16) {
+        vst1q_f32(out + j, vmulq_f32(vld1q_f32(acc + j), vinv));
+        vst1q_f32(out + j + 4, vmulq_f32(vld1q_f32(acc + j + 4), vinv));
+        vst1q_f32(out + j + 8, vmulq_f32(vld1q_f32(acc + j + 8), vinv));
+        vst1q_f32(out + j + 12, vmulq_f32(vld1q_f32(acc + j + 12), vinv));
+    }
+#elif defined(SE_HAVE_SSE)
+    __m128 vinv = _mm_set1_ps(inv);
+    for (; j + 15 < dim; j += 16) {
+        _mm_storeu_ps(out + j, _mm_mul_ps(_mm_loadu_ps(acc + j), vinv));
+        _mm_storeu_ps(out + j + 4, _mm_mul_ps(_mm_loadu_ps(acc + j + 4), vinv));
+        _mm_storeu_ps(out + j + 8, _mm_mul_ps(_mm_loadu_ps(acc + j + 8), vinv));
+        _mm_storeu_ps(out + j + 12, _mm_mul_ps(_mm_loadu_ps(acc + j + 12), vinv));
+    }
+#endif
+    for (; j < dim; j++)
+        out[j] = acc[j] * inv;
+}
+
+void se_l2_normalize(float *vec, uint32_t dim) {
+    double sum = 0.0;
+    for (uint32_t j = 0; j < dim; j++)
+        sum += (double)vec[j] * (double)vec[j];
+    if (sum <= 0.0)
+        return;
+    float inv = (float)(1.0 / sqrt(sum));
+    for (uint32_t j = 0; j < dim; j++)
+        vec[j] *= inv;
+}
+
+static se_status_t embed_ids_core(const se_model_t *model, se_scratch_t *sc, const uint32_t *ids,
+                                  size_t n, float *out, se_error_t *err,
+                                  volatile sig_atomic_t *cancelled) {
+    const uint32_t dim = model->meta.dim;
+    float *acc = sc->acc;
+    memset(acc, 0, (size_t)dim * sizeof(float));
+
+    const uint32_t drop_unk = (model->meta.unk_policy == SE_UNK_DROP);
+    const uint32_t unk_id = model->meta.unk_id;
+
+    size_t used = 0;
+    for (size_t i = 0; i < n; i++) {
+        if (((i & 255u) == 0) && cancelled && *cancelled) {
+            se_error_set(err, SE_ERR_INTERNAL, "operation cancelled");
+            return SE_ERR_INTERNAL;
+        }
+        uint32_t id = ids[i];
+        if (id >= model->meta.vocab_size) {
+            se_error_set(err, SE_ERR_INVALID_FORMAT, "token id %u is out of range", id);
+            return SE_ERR_INVALID_FORMAT;
+        }
+        if (drop_unk && id == unk_id)
+            continue;
+
+        if (i + SE_PREFETCH_DISTANCE < n) {
+            uint32_t next = ids[i + SE_PREFETCH_DISTANCE];
+            if (next < model->meta.vocab_size)
+                SE_PREFETCH(model->embeddings + (size_t)next * dim);
+        }
+
+        const float *row = model->embeddings + (size_t)id * dim;
+
+        add_row(acc, row, dim);
+        used++;
+    }
+
+    if (used == 0) {
+        if (model->meta.empty_policy == SE_EMPTY_RAISE) {
+            se_error_set(err, SE_ERR_EMPTY_INPUT, "input produced no usable tokens");
+            return SE_ERR_EMPTY_INPUT;
+        }
+        memset(out, 0, (size_t)dim * sizeof(float));
+        return SE_OK;
+    }
+
+    if (model->meta.normalization_type == SE_NORMALIZATION_L2) {
+        scale_copy(out, acc, dim, 1.0f);
+        se_l2_normalize(out, dim);
+    } else {
+        const float inv = 1.0f / (float)used;
+        scale_copy(out, acc, dim, inv);
+    }
+
+    return SE_OK;
+}
+
+se_status_t se_embed_ids(const se_model_t *model, se_scratch_t *sc, const uint32_t *ids,
+                         size_t n_ids, float *out, se_token_stats_t *stats, se_error_t *err,
+                         volatile sig_atomic_t *cancelled) {
+    if (n_ids > UINT32_MAX) {
+        se_error_set(err, SE_ERR_OOM, "too many token ids");
+        return SE_ERR_OOM;
+    }
+    stats->token_count = (uint32_t)n_ids;
+    stats->unk_count = 0;
+    stats->truncated = 0;
+    for (size_t i = 0; i < n_ids; i++) {
+        if (ids[i] == model->meta.unk_id)
+            stats->unk_count++;
+    }
+    return embed_ids_core(model, sc, ids, n_ids, out, err, cancelled);
+}
+
+se_status_t se_embed_one(const se_model_t *model, se_scratch_t *sc, const uint8_t *input,
+                         size_t input_len, uint32_t max_tokens, float *out, se_token_stats_t *stats,
+                         se_error_t *err, volatile sig_atomic_t *cancelled) {
+    se_status_t rc = se_tokenize(model, sc, input, input_len, max_tokens, stats, err, cancelled);
+    if (rc != SE_OK)
+        return rc;
+    return embed_ids_core(model, sc, sc->ids, stats->token_count, out, err, cancelled);
+}
+
+size_t se_model_warmup(const se_model_t *model) {
+    const volatile uint8_t *p = (const uint8_t *)model->map_base;
+    size_t n = model->map_size;
+    size_t page = 4096;
+#if !defined(_WIN32) && defined(_SC_PAGESIZE)
+    long sys_page = sysconf(_SC_PAGESIZE);
+    if (sys_page > 0)
+        page = (size_t)sys_page;
+#endif
+    volatile uint64_t sink = 0;
+    size_t touched = 0;
+
+#if defined(HAVE_MADVISE) && !defined(_WIN32)
+    (void)madvise(model->map_base, model->map_size, MADV_WILLNEED);
+#endif
+
+    for (size_t i = 0; i < n; i += page) {
+        sink += p[i];
+        touched++;
+    }
+    if (n)
+        sink += p[n - 1];
+
+    (void)sink;
+    return touched;
+}