static_embeddings 0.1.1 → 0.1.3

data/docs/MODEL_AUDIT.md

@@ -1,24 +1,37 @@
 # Model Audit
 
-This document records the trust and parity status of converted `.semb` models.
-A converted production model is accepted only after parity against the upstream
-Python `model2vec.StaticModel` implementation is recorded here.
+A converted model is trusted only after parity against upstream
+`model2vec.StaticModel` is recorded here. An audit record covers one `.semb`
+file **and** one runtime version: changing the tokenizer, normalizer, prefix
+window, pooling, output normalization or the oracle's corpus invalidates the
+runtime half of it even though the file bytes are untouched.
 
 ## potion-retrieval-32m
 
-Status: audited / accepted
+| runtime | parity | note |
+|---|---|---|
+| 0.1.1 | `parity OK`, recorded below | superseded |
+| 0.1.2 | **not re-run** | superseded before release |
+| 0.1.3 | `parity OK`, recorded below | current |
+
+0.1.2 changed `is_control()`, which changes token ids for any input containing
+`U+007F`. `StaticEmbeddings::Reference` cannot settle whether the new behaviour
+matches HuggingFace, because it is an implementation twin of the C runtime
+written in this repository. The 0.1.3 release candidate was checked against a
+fresh upstream `model2vec.StaticModel` oracle that includes DEL, control
+characters, Unicode, OOV, long-word and truncation rows.
 
 Source model:
 
 - Hugging Face repository: `minishlab/potion-retrieval-32M`
+- Hugging Face snapshot: `6fc8051fab2a1e0ee76689cf08c853792ac285e7`
 - Oracle implementation: `model2vec.StaticModel.from_pretrained`
 - Python package: `model2vec 0.9.0`
 - Oracle file: `tmp/model2vec_oracle.json`
-- Oracle rows: `28`
+- Oracle rows in this recorded run: `31`
 - Oracle dimension: `512`
 - Oracle max length: `512`
-- Runtime checked with: `static_embeddings 0.1.1`
-- Runtime source note: re-run parity after changing tokenizer, normalizer, prefix-window, pooling, or output-normalization code.
+- Runtime at time of this record: `static_embeddings 0.1.3`
 
 Converted `.semb`:
 
@@ -31,6 +44,9 @@ Converted `.semb`:
 Parity command:
 
 ```bash
+python tools/model2vec_oracle.py minishlab/potion-retrieval-32M \
+  --out tmp/model2vec_oracle.json
+
 bundle exec rake parity \
   MODEL="$HOME/.cache/static_embeddings/models/potion-retrieval-32m.semb" \
   ORACLE=tmp/model2vec_oracle.json
@@ -39,8 +55,8 @@ bundle exec rake parity \
 Parity result:
 
 ```text
-rows=28
-id_rows_checked=28
+rows=31
+id_rows_checked=31
 min_cosine=0.9999999999989528
 max_abs_all=2.980232238769531e-07
 token_id_failures=[]
@@ -57,5 +73,8 @@ Decision:
 - Unicode normalization behavior: pass
 - Unknown-word behavior: pass
 - Long input / truncation behavior: pass
+- DEL / control-character behavior: pass
 
-This converted model is accepted for runtime and benchmark use.
+Accepted for runtime and benchmark use **under 0.1.3**. The `.semb` file is
+unchanged and its SHA256 still matches; the runtime half of the record was
+refreshed after the DEL fix.

data/docs/PERFORMANCE.md

@@ -1,60 +1,136 @@
 # Performance budget
 
-Absolute latency numbers are not comparable across models: `potion-base-2M`
-and `potion-retrieval-32M` differ by roughly 8x in row width, so swapping the
-model would read as a regression. The budget is therefore normalised.
+Absolute latency is not comparable across models: `potion-base-2M` and
+`potion-retrieval-32M` differ by roughly 8x in row width, so swapping the model
+would read as a regression. The budget is therefore normalised.
 
-Run `ruby tools/benchmark.rb [model.semb]`.
+```bash
+ruby tools/benchmark.rb [model.semb]
+```
 
 ## Metrics tracked
 
 | Metric | Unit | Why |
 |---|---|---|
-| tokenization | ns / input byte | independent of `dim`; catches normalizer regressions |
+| tokenization | ns / processed byte | independent of `dim`; catches normalizer regressions |
 | pooling | ns / token / 100 dims | comparable across models |
 | single embed | µs / call | the latency a query pays |
 | batch throughput | texts/s, tokens/s | indexing capacity |
 | RSS overhead | bytes beyond the mapped file | must stay O(scratch), not O(model) |
-| cold vs warm | first query after a cold page cache | mmap means the first pass faults in the matrix |
 | time to first query | ms from `load` to first vector | what a Puma `before_fork` or a serverless cold start pays |
+| C allocation stats | bytes and counts by native category | optional build, explains heap churn |
 
-`samples/cold_start.rb` measures the last row. It is single shot on purpose:
-`load` walks the whole vocabulary hash table and both tries exactly once, and
-`warmup!` only faults pages in once. Drop the page cache first
-(`sudo purge` on macOS, `echo 3 > /proc/sys/vm/drop_caches` on Linux) or the
-output will label itself a warm start.
+## Where the time actually goes
+
+Tokenization dominates; pooling is a handful of row reads and adds. That is why
+the tokenizer is the file to profile, and why SIMD in the pooling loop is not
+where the wins are — that loop is memory-bound on random row lookups and the
+compiler already vectorises it.
+
+Out-of-vocabulary text is the main cliff. In-vocabulary words hit the
+vocabulary hash directly; everything else falls through to trie-driven subword
+splitting, which on a synthetic all-OOV corpus measured several times the
+per-text cost. Non-English input on an English model pays this on top of the
+`[UNK]` quality problem.
+
+## `f16` is a storage trade-off
+
+`format: :f16` halves the bytes a top-k scan streams and doubles the decode
+work. Which one wins is a property of the machine, and this repository's own
+sample runs disagree with each other — same corpus, same `k`, native decode
+kernel in both cases:
+
+```text
+                     f32        f16
+x86-64, f16c       3.00 ms    1.65 ms    f16 1.8x faster
+M1 Pro, neon-fp16  2.29 ms    2.88 ms    f16 1.3x slower
+```
+
+Neither ratio transfers. Confirm `StaticEmbeddings.simd_backend`, then measure
+on the hardware you will run on. On the lookup-table fallback `f16` is usually
+slower than `f32`.
+
+## C allocation counters
+
+Compiling with `STATIC_EMBEDDINGS_ALLOC_STATS=1` wraps the runtime's own
+allocations and records bytes and counts per category. Default builds do not
+define the internal methods and do not pay for the counters.
+
+```bash
+STATIC_EMBEDDINGS_ALLOC_STATS=1 samples/run_all.sh
+```
 
-Every other sample under `samples/` runs a hot loop and measures steady state.
-`warmup_hot_path.rb` in particular re-touches pages that are already resident;
-its `mapped_range_gb_per_sec` is address-range coverage, not memory bandwidth,
-because warmup reads one byte per page.
+The harness forces `rake clobber compile` in this mode, because an existing
+mkmf Makefile otherwise keeps the previous `DEFS`, and it fails the run if the
+instrumented methods did not load. Samples then print
+`c_alloc.<category>.<metric>` lines.
 
-## CI status
+Two limits worth stating. The counters cover only allocations made through the
+instrumented wrappers, so Ruby object heap, allocator fragmentation and mmap
+residency are all outside them — use Instruments, heaptrack, Massif or `vmmap`
+for those. And the instrumentation is not free: about 13% on `tokenize`, 3% on
+`embed`, within noise on `embed_batch`. A run captured with it is not
+comparable to one captured without it.
 
-The CI benchmark job is informational. Treat regression gates as meaningful only when a checked-in baseline is compared on the same runner class. Absolute thresholds on shared CI hardware produce noise, not signal.
+## Running the benchmarks
+
+`bundle exec rake benchmark`, or one at a time with `rake benchmark:core_paths`
+and `rake benchmark:gvl_threshold`. Generated text inputs are
+`valid_encoding?`-primed before timing, so these are cached-coderange hot-path
+numbers. Every row carries `spread_pct`, and a difference smaller than that is
+not a result. See `docs/BENCHMARKING.md`.
 
 ## Benchmark hygiene
 
-Two failure modes have bitten this repository already and are worth stating:
+Three failure modes have bitten this repository already.
 
-- **Logical vs processed bytes.** With truncation active, a 3 MB document is
-  tokenized only until `max_tokens` is reached. Any "MB/s of input" figure
-  divides by bytes that were never read. Samples that still print one label it
-  `logical_input_mb_per_sec` and set `truncation_active`.
-- **Resident working sets.** Embedding the same text, or the same frozen id
-  array, in a loop keeps a tiny slice of the matrix in cache and overstates
-  throughput. `random_pooling_hot_path.rb` rotates through many id sets and
-  prints `touched_matrix_mb` so the reader can check it exceeds the last level
-  cache.
+**Logical vs processed bytes.** With truncation active a 3 MB document is
+tokenized only until `max_tokens` is reached, so any "MB/s of input" figure
+computed from the string length divides by bytes that were never read. Samples
+print both `logical_input_mb_per_sec` and a measured
+`processed_input_mb_per_sec`; quote the second. `tools/benchmark.rb` refuses to
+print a ns/byte figure at all when truncation makes the two differ.
 
-`GC.disable` is off by default in the sample harness: it does not stop an
-already started incremental cycle from sweeping, so zeroed GC counters used to
-appear next to profiles full of `gc_sweep`. Set `GC_DISABLE=1` deliberately and
-read `gc_disabled=` before quoting `gc_delta`.
+**Resident working sets.** Embedding the same text, or the same frozen id array,
+in a loop keeps a tiny slice of the matrix in cache and overstates throughput.
+`random_pooling_hot_path.rb` rotates through many id sets and prints
+`touched_matrix_mb` so you can check it exceeds the last level cache.
 
-## Where the time actually goes
+**GC state.** `GC.disable` does not stop an already started incremental cycle
+from sweeping, so zeroed GC counters used to appear next to profiles full of
+`gc_sweep`. It is off by default in the harness; set `GC_DISABLE=1`
+deliberately and read `gc_disabled=` before quoting `gc_delta`.
+
+## Two costs that are not the tokenizer
+
+`embed` cannot start until Ruby has confirmed the `String` is valid UTF-8, and
+for a `String` whose coderange has not been computed that is an O(bytes) scan
+which runs before the prefix window is chosen. On a 3 MB freshly read document
+it dominated: 442 µs against 71 µs for the same string once Ruby had cached the
+answer. Reuse the `String`, call `valid_encoding?` outside the hot path, or pass
+`validate_encoding: :prefix` to bound validation the same way tokenizing is
+bounded — 78 µs, at the cost of not inspecting bytes past the window.
+
+`StaticEmbeddings.verify` hashes the artifact and is deliberately not on the
+load path. It streams in 1 MiB chunks: on a 62 MB model, 249 ms and 30 MB of
+peak RSS. Reading the file whole and duplicating it to zero the checksum field
+cost 433 ms and 280 MB.
+
+## Cold start
+
+`samples/cold_start.rb` measures `load`, first embed and `warmup!` in one shot,
+because each of those happens exactly once: `load` walks the vocabulary hash and
+both tries, and `warmup!` faults pages in. Drop the page cache first
+(`sudo purge` on macOS, `echo 3 > /proc/sys/vm/drop_caches` on Linux) or the
+output labels itself a warm start.
+
+Every other sample runs a hot loop and measures steady state. `warmup_hot_path.rb`
+in particular re-touches resident pages; its `mapped_range_gb_per_sec` is
+address-range coverage, not memory bandwidth, because warmup reads one byte per
+page.
+
+## CI
 
-For static embeddings, tokenization dominates: pooling is a handful of row
-reads and adds. That is why the tokenizer is the file to profile, and why
-SIMD in the pooling loop is not where the wins are — the loop is memory-bound
-on random row lookups, and the compiler already vectorises it.
+The benchmark job is informational. Absolute thresholds on shared runners
+produce noise, not signal; a regression gate is only meaningful against a
+checked-in baseline on the same runner class.

data/ext/static_embeddings/extconf.rb

@@ -24,6 +24,9 @@ end
 
 $srcs = %w[
   static_embeddings.c
+  se_alloc_stats.c
+  se_f16.c
+  se_topk.c
   se_format.c
   se_unicode.c
   se_tokenizer.c
@@ -40,5 +43,6 @@ end
 
 $CFLAGS += " -O3 -std=gnu99 -fvisibility=hidden"
 $CFLAGS += " -Wall -Wextra -Wno-unused-parameter"
+$defs << "-DSE_ENABLE_ALLOC_STATS=1" if ENV["STATIC_EMBEDDINGS_ALLOC_STATS"] == "1"
 
 create_makefile("static_embeddings/static_embeddings")

data/ext/static_embeddings/se_alloc_stats.c

@@ -0,0 +1,244 @@
+#include "se_internal.h"
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+
+static const char *const se_alloc_category_names[SE_ALLOC_CATEGORY_COUNT] = {
+    "unknown",    "scratch",     "batch_input",      "batch_output",
+    "batch_stats", "batch_index", "token_ids",       "topk_query",
+    "topk_best",  "topk_matrix_copy", "format_validate", "model_file"};
+
+const char *se_alloc_category_name(se_alloc_category_t category) {
+    if ((unsigned)category >= SE_ALLOC_CATEGORY_COUNT)
+        return se_alloc_category_names[SE_ALLOC_UNKNOWN];
+    return se_alloc_category_names[category];
+}
+
+#if SE_ENABLE_ALLOC_STATS
+
+#define SE_ALLOC_STATS_MAGIC 0x5EA110C5u
+
+typedef union {
+    struct {
+        uint32_t magic;
+        uint32_t category;
+        size_t size;
+    } h;
+    long double align_long_double;
+    void *align_ptr;
+    uint64_t align_u64;
+} se_alloc_header_t;
+
+static se_alloc_stats_t se_alloc_stats[SE_ALLOC_CATEGORY_COUNT];
+
+static se_alloc_category_t normalize_category(se_alloc_category_t category) {
+    if ((unsigned)category >= SE_ALLOC_CATEGORY_COUNT)
+        return SE_ALLOC_UNKNOWN;
+    return category;
+}
+
+#if defined(__GNUC__) || defined(__clang__)
+static size_t atomic_add_size(size_t *ptr, size_t value) {
+    return __sync_add_and_fetch(ptr, value);
+}
+
+static size_t atomic_sub_size(size_t *ptr, size_t value) {
+    return __sync_sub_and_fetch(ptr, value);
+}
+
+static void atomic_inc_size(size_t *ptr) {
+    __sync_fetch_and_add(ptr, (size_t)1);
+}
+
+static size_t atomic_load_size(const size_t *ptr) {
+    return __sync_fetch_and_add((size_t *)ptr, (size_t)0);
+}
+
+static void atomic_store_size(size_t *ptr, size_t value) {
+    __sync_lock_test_and_set(ptr, value);
+}
+
+static int atomic_cas_size(size_t *ptr, size_t old_value, size_t new_value) {
+    return __sync_bool_compare_and_swap(ptr, old_value, new_value);
+}
+#else
+static size_t atomic_add_size(size_t *ptr, size_t value) {
+    *ptr += value;
+    return *ptr;
+}
+
+static size_t atomic_sub_size(size_t *ptr, size_t value) {
+    *ptr -= value;
+    return *ptr;
+}
+
+static void atomic_inc_size(size_t *ptr) {
+    (*ptr)++;
+}
+
+static size_t atomic_load_size(const size_t *ptr) {
+    return *ptr;
+}
+
+static void atomic_store_size(size_t *ptr, size_t value) {
+    *ptr = value;
+}
+
+static int atomic_cas_size(size_t *ptr, size_t old_value, size_t new_value) {
+    if (*ptr != old_value)
+        return 0;
+    *ptr = new_value;
+    return 1;
+}
+#endif
+
+static void record_peak(se_alloc_category_t category, size_t current) {
+    size_t peak = atomic_load_size(&se_alloc_stats[category].peak_bytes);
+    while (current > peak) {
+        if (atomic_cas_size(&se_alloc_stats[category].peak_bytes, peak, current))
+            return;
+        peak = atomic_load_size(&se_alloc_stats[category].peak_bytes);
+    }
+}
+
+static void record_alloc(se_alloc_category_t category, size_t bytes) {
+    se_alloc_stats_t *stats = &se_alloc_stats[category];
+    size_t current = atomic_add_size(&stats->current_bytes, bytes);
+    atomic_add_size(&stats->total_allocated_bytes, bytes);
+    atomic_inc_size(&stats->alloc_count);
+    record_peak(category, current);
+}
+
+static void record_realloc(se_alloc_category_t old_category, size_t old_bytes,
+                           se_alloc_category_t new_category, size_t new_bytes) {
+    se_alloc_stats_t *old_stats = &se_alloc_stats[old_category];
+    se_alloc_stats_t *new_stats = &se_alloc_stats[new_category];
+    atomic_sub_size(&old_stats->current_bytes, old_bytes);
+    atomic_add_size(&old_stats->total_freed_bytes, old_bytes);
+    atomic_inc_size(&old_stats->free_count);
+
+    size_t current = atomic_add_size(&new_stats->current_bytes, new_bytes);
+    atomic_add_size(&new_stats->total_allocated_bytes, new_bytes);
+    atomic_inc_size(&new_stats->alloc_count);
+    atomic_inc_size(&new_stats->realloc_count);
+    record_peak(new_category, current);
+}
+
+static void record_free(se_alloc_category_t category, size_t bytes) {
+    se_alloc_stats_t *stats = &se_alloc_stats[category];
+    atomic_sub_size(&stats->current_bytes, bytes);
+    atomic_add_size(&stats->total_freed_bytes, bytes);
+    atomic_inc_size(&stats->free_count);
+}
+
+static se_alloc_header_t *header_from_ptr(void *ptr) {
+    return ((se_alloc_header_t *)ptr) - 1;
+}
+
+void *se_alloc_stats_malloc(se_alloc_category_t category, size_t bytes) {
+    size_t total = 0;
+    if (!se_checked_add_size(sizeof(se_alloc_header_t), bytes, &total))
+        return NULL;
+
+    se_alloc_header_t *header = (se_alloc_header_t *)malloc(total);
+    if (!header)
+        return NULL;
+
+    category = normalize_category(category);
+    header->h.magic = SE_ALLOC_STATS_MAGIC;
+    header->h.category = (uint32_t)category;
+    header->h.size = bytes;
+    record_alloc(category, bytes);
+    return (void *)(header + 1);
+}
+
+void *se_alloc_stats_calloc(se_alloc_category_t category, size_t count, size_t elem_size) {
+    size_t bytes = 0;
+    if (!se_checked_mul_size(count, elem_size, &bytes))
+        return NULL;
+
+    void *ptr = se_alloc_stats_malloc(category, bytes);
+    if (ptr)
+        memset(ptr, 0, bytes);
+    return ptr;
+}
+
+void *se_alloc_stats_realloc(se_alloc_category_t category, void *ptr, size_t bytes) {
+    if (!ptr)
+        return se_alloc_stats_malloc(category, bytes);
+
+    size_t total = 0;
+    if (!se_checked_add_size(sizeof(se_alloc_header_t), bytes, &total))
+        return NULL;
+
+    se_alloc_header_t *old_header = header_from_ptr(ptr);
+    if (old_header->h.magic != SE_ALLOC_STATS_MAGIC)
+        abort();
+
+    se_alloc_category_t old_category = normalize_category((se_alloc_category_t)old_header->h.category);
+    size_t old_bytes = old_header->h.size;
+    se_alloc_header_t *new_header = (se_alloc_header_t *)realloc(old_header, total);
+    if (!new_header)
+        return NULL;
+
+    category = normalize_category(category);
+    new_header->h.magic = SE_ALLOC_STATS_MAGIC;
+    new_header->h.category = (uint32_t)category;
+    new_header->h.size = bytes;
+    record_realloc(old_category, old_bytes, category, bytes);
+    return (void *)(new_header + 1);
+}
+
+void se_alloc_stats_free(void *ptr) {
+    if (!ptr)
+        return;
+
+    se_alloc_header_t *header = header_from_ptr(ptr);
+    if (header->h.magic != SE_ALLOC_STATS_MAGIC)
+        abort();
+
+    se_alloc_category_t stored_category =
+        normalize_category((se_alloc_category_t)header->h.category);
+    size_t bytes = header->h.size;
+    header->h.magic = 0;
+    record_free(stored_category, bytes);
+    free(header);
+}
+
+void se_alloc_stats_reset(void) {
+    for (size_t i = 0; i < SE_ALLOC_CATEGORY_COUNT; i++) {
+        se_alloc_stats_t *stats = &se_alloc_stats[i];
+        size_t current = atomic_load_size(&stats->current_bytes);
+        atomic_store_size(&stats->peak_bytes, current);
+        atomic_store_size(&stats->total_allocated_bytes, 0);
+        atomic_store_size(&stats->total_freed_bytes, 0);
+        atomic_store_size(&stats->alloc_count, 0);
+        atomic_store_size(&stats->realloc_count, 0);
+        atomic_store_size(&stats->free_count, 0);
+    }
+}
+
+void se_alloc_stats_snapshot(se_alloc_stats_t out[SE_ALLOC_CATEGORY_COUNT]) {
+    for (size_t i = 0; i < SE_ALLOC_CATEGORY_COUNT; i++) {
+        out[i].current_bytes = atomic_load_size(&se_alloc_stats[i].current_bytes);
+        out[i].peak_bytes = atomic_load_size(&se_alloc_stats[i].peak_bytes);
+        out[i].total_allocated_bytes =
+            atomic_load_size(&se_alloc_stats[i].total_allocated_bytes);
+        out[i].total_freed_bytes = atomic_load_size(&se_alloc_stats[i].total_freed_bytes);
+        out[i].alloc_count = atomic_load_size(&se_alloc_stats[i].alloc_count);
+        out[i].realloc_count = atomic_load_size(&se_alloc_stats[i].realloc_count);
+        out[i].free_count = atomic_load_size(&se_alloc_stats[i].free_count);
+    }
+}
+
+#else
+
+void se_alloc_stats_reset(void) {
+}
+
+void se_alloc_stats_snapshot(se_alloc_stats_t out[SE_ALLOC_CATEGORY_COUNT]) {
+    memset(out, 0, sizeof(se_alloc_stats_t) * SE_ALLOC_CATEGORY_COUNT);
+}
+
+#endif