static_embeddings 0.1.5 → 1.5.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fb1d3530fba4fad467251429797fdcaae527b5c6c18413e8847002e0c3f3b5e4
-  data.tar.gz: f02bb4c129449803a57b08bf9b7186af08a6d21edee1e68d18178b248b855e0b
+  metadata.gz: 550531f7573a8d05e8d5a3a1ae0fa301c01d77b238573ae11834797e8ac44a0f
+  data.tar.gz: 57f487c729296e3a858604dc64aa146aa6dd6e372a45961d90a61564e3c1b117
 SHA512:
-  metadata.gz: cec0a3959cb3f093c7bb2b06213b4c7e983986466a115eb79e0b045a5b52fdda9797be41365db41a74d85bb52c384c147d68d3d84c503fef10c122bba7b72333
-  data.tar.gz: e99039606af3de0106e1ac26efaa78a06f09acbd9c2faaae9b4ca17339bdeb4830ba922bef84dc141317898556d09b1ec41cf1ebfebcce5e141d877c3fc34598
+  metadata.gz: 52162c96d458dffaaec8498cd712678b4727b29ddaa66eadf7834893113a6bd05f636cc47a2f84d969f76eb4651fe57664212fdb4ac2925449827b9d3a69a9bf
+  data.tar.gz: 115a0b0f3ff9d82cea099fc0ee24385034d8bcc66ca4ff9337f2213608ee4d611215b6d3400fe39aecf562cf201f5f5044eab7f9863fbdc813362c0af0eb7739

data/CHANGELOG.md

@@ -1,5 +1,55 @@
 # Changelog
 
+## 1.5.6
+
+Multi-source WordPiece release. The native `.semb` v3 runtime is unchanged; new
+source families are normalized offline into the same audited
+`BERT_WORDPIECE_V1 -> lookup -> mean -> NONE/L2` contract. Model identity is
+chosen at `load_model`, not per `embed` call.
+
+### Added
+
+- **Model2Vec and Sentence Transformers importers.** A Model2Vec directory and a
+  Sentence Transformers `StaticEmbedding` directory (`modules.json` + exactly
+  one module path) compile to the same canonical data. The ST importer is
+  fail-closed and confines the module path to the source root with `realpath`.
+- **Source-faithful policies.** Model2Vec uses `UNK_DROP`, normalization from its
+  config and a 512-token default. Sentence Transformers `StaticEmbedding` uses
+  `UNK_INCLUDE`, `NORMALIZATION_NONE`, unlimited tokens and
+  `add_special_tokens=false`.
+- **`--dimensions N`.** Matryoshka prefix slicing happens while streaming the
+  embedding matrix; the runtime still loads an ordinary fixed-dimension `.semb`.
+- **`--max-tokens unlimited`.** Convert-time `0` remains an alias; runtime
+  `max_tokens: 0` is still rejected and `false` means unlimited.
+- **Russian retrieval fixture.** `tools/eval_retrieval.rb` and the checked-in
+  Russian FAQ corpus provide a small domain sanity check for the multilingual
+  WordPiece model; it is not presented as a general benchmark.
+
+### Changed
+
+- **Offline Ruby conversion was simplified.** Stateful `Converter` and the
+  behavior-heavy `CanonicalModel` were replaced by import functions, immutable
+  canonical data, pure provenance/meta transforms and a streaming format writer.
+  Writer/hash/trie/verifier code is split from runtime format constants, so plain
+  `require "static_embeddings"` does not load conversion code.
+- The Ruby `Reference` twin follows canonical UNK/normalization/max-token
+  policies instead of re-deriving Model2Vec defaults.
+- Model2Vec `normalize` now defaults to `false` when the key is absent, matching
+  the pinned upstream behavior. Unknown source families are rejected rather than
+  guessed from `tokenizer.json + model.safetensors`.
+- Header integer writes validate their unsigned range instead of silently
+  wrapping oversized Ruby integers.
+- `Model#provenance` is parsed once and cached.
+
+Local WordPiece conversions requiring **no C change**: `potion-base-8M`,
+`potion-science-32M`, `static-retrieval-mrl-en-v1` at 1024/512 and
+`static-similarity-mrl-multilingual-v1` at 512/256. The retrieval model has a
+recorded `SentenceTransformer.encode` oracle (436/436). Records are in
+`docs/MODEL_AUDIT.md`.
+
+SentencePiece/Unigram is intentionally **not** part of 1.5.6; it remains a
+future runtime capability rather than shipping an unaudited v4 path.
+
 ## 0.1.5
 
 Correctness/parity release. The native runtime architecture is unchanged, but

data/README.md

@@ -1,11 +1,11 @@
 # static_embeddings
 
-A Ruby runtime for converted Model2Vec / potion static embedding models. A model
+A Ruby runtime for converted Model2Vec and Sentence Transformers static embedding models. A model
 is converted once into a local `.semb` file and loaded through a small C
 extension. No ONNX Runtime, no Rust or Python at runtime, no network access, one
 mmap-able file, binary float32 output.
 
-The current production target is `minishlab/potion-retrieval-32M`.
+The reference production target remains `minishlab/potion-retrieval-32M`; 1.5.6 also supports audited WordPiece `StaticEmbedding` sources.
 
 ```ruby
 require "static_embeddings"
@@ -49,6 +49,21 @@ bundle exec ruby -Ilib exe/static_embeddings convert ./potion-retrieval-32M \
   --id potion-retrieval-32m
 ```
 
+Sentence Transformers static models (`modules.json` + one `StaticEmbedding`
+module) use the same command. Matryoshka prefix-slice is a convert-time flag,
+not a runtime `dim:`:
+
+```bash
+bundle exec ruby -Ilib exe/static_embeddings convert ./static-retrieval-mrl-en-v1 \
+  --id static-retrieval-mrl-en-v1-512 \
+  --dimensions 512 \
+  --trained-mrl-dims 1024,512,256,128,64,32
+```
+
+`--max-tokens unlimited` bakes no token cap (Sentence Transformers default).
+`--max-tokens 0` is a convert-time alias; `embed(max_tokens: 0)` is still
+rejected. Query and documents must use the same loaded `.semb`.
+
 The result lands in `~/.cache/static_embeddings/models/potion-retrieval-32m.semb`.
 Inspect or verify it with the `inspect` and `verify` subcommands, then load it:
 

data/docs/ARCHITECTURE.md

@@ -8,10 +8,10 @@ HuggingFace / Model2Vec files          offline, once, on your machine
   model.safetensors
         |
         v
-  Converter (pure Ruby, strict)   <--- all parsing, all validation, all
-        |                              Unicode table generation happens here
+  import -> canonical data -> writer   <--- pure Ruby, offline, strict
+        |                                    all source parsing/validation here
         v
-  model.semb                       <--- flat, versioned, mmap-able
+  model.semb                            <--- flat, versioned, mmap-able
         |
         v
   C runtime                        <--- mmap + bounds checks + tokenize +
@@ -24,6 +24,15 @@ Everything expensive, fragile or security-sensitive about reading third-party
 model files happens once, offline, in a language where it is easy to get
 right. What remains in C is a bounds-checked mmap and three loops.
 
+Only the offline import layer knows HuggingFace layouts. Model2Vec and Sentence
+Transformers `StaticEmbedding` sources are reduced to immutable canonical data:
+tokens, streamed matrix, tokenizer metadata, runtime policies and provenance.
+Pure transforms then build the v3 header/provenance and the existing WordPiece
+writer emits the artifact. A new WordPiece + mean-pooling source therefore does
+not require a C change. New tokenizer families remain explicit future
+capabilities and require a separately audited format/runtime change rather than
+source-name special cases.
+
 The C side is `se_format.c` (mmap and validation), `se_tokenizer.c`,
 `se_embed.c`, `se_unicode.c`, `se_f16.c` (half-precision codec and kernels),
 `se_topk.c`, `se_alloc_stats.c` (optional allocation counters), and
@@ -336,8 +345,8 @@ chunks -> .semb vectors --/
 1. **Which potion models actually match `BERT_WORDPIECE_V1`.** The runtime now
    implements the exact `normalized: false` extraction semantics needed by the
    five standard BERT added tokens, but still rejects arbitrary AddedVocabulary,
-   BPE and Unigram profiles. Each source model still needs an audit before it is
-   treated as compatible.
+   BPE and Unigram profiles. Type A (WordPiece + mean) models convert without a
+   C change; each source still needs an audit before it is treated as compatible.
 2. **Russian.** Distilling a multilingual teacher into a WordPiece vocabulary
    we control keeps the pure-C path, but skips the Tokenlearn pre-training
    that gives the published potion models their quality. That gap has to be

data/docs/MODEL_AUDIT.md

@@ -119,3 +119,113 @@ synthetic `tiny-wordpiece` fixture: `434/434`, all four failure lists empty,
 `min_cosine=0.9999999999999988`, `max_abs_all=5.960464477539063e-08`. That job
 proves the fixture loads in `StaticModel.from_pretrained` and that the checker
 contracts hold; it is not a potion audit.
+
+## 1.5.6 Patch 1 proof
+
+Architecture: Model2Vec / Sentence Transformers import functions → immutable canonical data →
+existing WordPiece `.semb` v3 writer. C runtime unchanged.
+
+### potion-retrieval-32M reconvert
+
+Official snapshot `6fc8051fab2a1e0ee76689cf08c853792ac285e7` ships a Sentence
+Transformers `modules.json` (`StaticEmbedding` + `Normalize`). Detection uses
+`config.json` `model_type=model2vec` first, so this stays UNK_DROP / L2 / 512.
+
+New file `potion-retrieval-32m-reconvert.semb` vs existing
+`potion-retrieval-32m-v3.semb`: vocab/hash/embeddings/norm_tables/tries
+byte-identical. 10-text corpus `max_abs=0`. Provenance JSON differs (new keys).
+
+### static-retrieval-mrl-en-v1 vs SentenceTransformer.encode
+
+Oracle: `tools/st_oracle.py` + `tools/check_st_parity.rb`,
+`sentence-transformers 6.0.1`, `add_special_tokens=false`,
+`normalize_embeddings=false`, 436 rows including `all-unk:private-use` and
+`long:known-words-600`.
+
+```text
+1024: 436/436  min_cosine=0.999999999999999  max_abs_all=7.62939453125e-06
+512:  436/436  min_cosine=0.9999999999999989 max_abs_all=7.62939453125e-06
+```
+
+512 compared as prefix of the 1024-d encode vectors. Ruby Reference matched C
+separately. Diagnostic `StaticEmbedding.forward` matched encode on this model.
+
+### UNK and max_tokens on the real models
+
+- potion-retrieval-32M `🧬 🧬 🧬` → UNK ids dropped → zero vector; 600×`hello`
+  → `truncated=true pooled=512`.
+- static-retrieval-mrl-en-v1-1024 same emoji string → three UNK ids pooled
+  (non-zero, mean of UNK rows); 600×`hello` → `truncated=false pooled=600`.
+- Private-use codepoints are BertNormalizer `Co` and become empty, not UNK.
+
+## 1.5.6 Type A conversions
+
+These records prove the offline import layer maps extra WordPiece sources onto
+the existing runtime. They are **not** a replacement for the
+potion-retrieval-32m Python oracle above. `static-retrieval-mrl-en-v1` has the
+separate `SentenceTransformer.encode` oracle recorded above; the multilingual
+similarity artifact was checked against the Ruby `.semb` Reference twin and the
+small Russian retrieval fixture, not claimed as a general upstream retrieval
+benchmark.
+
+All conversions used `static_embeddings 1.5.6` on arm64-darwin24.
+
+### minishlab/potion-base-8M
+
+- Snapshot: `bf8b056651a2c21b8d2565580b8569da283cab23`
+- Layout: Model2Vec (`UNK_DROP`, L2, `max_tokens` 512)
+- `.semb`: `potion-base-8m.semb` `dim=256` `vocab=29528` `33346200` bytes
+  SHA256 `8abd8d1f26511959e14ca26ec4dcacad1d3db9b4b66b2e67683fb2d06074f490`
+- Native C matched Ruby Reference on short English/OOV/empty texts (`max_abs < 1e-5`)
+- `model2vec 0.9.0` / `tokenizers 0.23.1` oracle: 434/434, vectors_checked=432,
+  intentional character-pretruncate deviations=2, min_cosine=0.9999999999999989,
+  max_abs_all=2.98e-08
+
+### minishlab/potion-science-32M
+
+- Snapshot: `7366079845507de14a4330007cdfa01bb92bca52`
+- Layout: Model2Vec (`UNK_DROP`, L2, `max_tokens` 512)
+- `.semb`: `potion-science-32m.semb` `dim=256` `vocab=124428` `140353240` bytes
+  SHA256 `e968f93b89d59d9c63c6a8152495283d1138144e61ed5c08d306c21c00b7cdb1`
+- Native C matched Ruby Reference on English and Russian snippets (`max_abs < 1e-5`)
+- `model2vec 0.9.0` / `tokenizers 0.23.1` oracle: 434/434, vectors_checked=432,
+  intentional character-pretruncate deviations=2, min_cosine=0.9999999999999986,
+  max_abs_all=2.98e-08
+
+### sentence-transformers/static-retrieval-mrl-en-v1
+
+- Snapshot: `f60985c706f192d45d218078e49e5a8b6f15283a`
+- Layout: Sentence Transformers StaticEmbedding (`UNK_INCLUDE`, no L2, unlimited)
+- `static-retrieval-mrl-en-v1-1024.semb` `dim=1024` `vocab=30522` `128186264` bytes
+  SHA256 `f316bb07348503418ed0f6d897ddbf7b44422d2077ca4860518f31f328610946`
+- `static-retrieval-mrl-en-v1-512.semb` `--dimensions 512` `65677208` bytes
+  SHA256 `efbb43c14def793d0bbb42ef591a0951caed23a49cdd94dcf72312569d1b7cff`
+- Prefix of a 1024-d vector matched the 512-d artifact on in-vocabulary English
+  (valid because this source does not L2-normalize)
+- Native C matched Ruby Reference on short texts
+
+### sentence-transformers/static-similarity-mrl-multilingual-v1
+
+- Snapshot: `b68f4122911bcffcd6e1f695f2d99cd6788972d8`
+- Layout: Sentence Transformers StaticEmbedding (`UNK_INCLUDE`, no L2, unlimited)
+- `static-similarity-mrl-multilingual-v1-512.semb` `--dimensions 512` `vocab=105879`
+  `227983824` bytes SHA256 `8d9a63d20ee23468ae2fd2584ef3f0ec75e720e68410e416157aaf2b873e110e`
+- `static-similarity-mrl-multilingual-v1-256.semb` `--dimensions 256` `119563728` bytes
+  SHA256 `2e9e689bd3afd18ed50a77f0f8842b4b5a5be39bfad75e3fa24ad99b478de068`
+- Native C matched Ruby Reference on Russian text (`max_abs 9.1e-7`)
+
+This model is published as similarity, not retrieval. `tools/eval_retrieval.rb`
+on `test/fixtures/russian_faq_eval.json` (10 queries / 10 docs, cosine@10,
+arm64-darwin24, 1.5.6):
+
+| model | dim | MRR | nDCG@10 |
+|---|---:|---:|---:|
+| static-similarity-mrl-multilingual-v1-256 | 256 | 0.875 | 0.906 |
+| static-similarity-mrl-multilingual-v1-512 | 512 | 0.850 | 0.889 |
+| static-retrieval-mrl-en-v1-512 | 512 | 0.792 | 0.842 |
+| potion-retrieval-32m | 512 | 0.733 | 0.799 |
+| potion-base-8m | 256 | 0.712 | 0.780 |
+
+Hit@10 was 1.0 for every model on this tiny labeled set. That is a domain sanity
+check, not a public retrieval benchmark, and it does not make the similarity
+model a hard-coded Russian default.

data/lib/static_embeddings/bert_wordpiece.rb

@@ -0,0 +1,191 @@
+require "static_embeddings/errors"
+require "static_embeddings/format/constants"
+
+module StaticEmbeddings
+  module BertWordPiece
+    TOKENIZER_PROFILE = "BERT_WORDPIECE_V1"
+    ALLOWED_NORMALIZER_KEYS = %w[type clean_text handle_chinese_chars strip_accents lowercase].freeze
+    STANDARD_SPECIAL_TOKENS = {
+      "[PAD]" => Format::ADDED_PAD,
+      "[UNK]" => Format::ADDED_UNK,
+      "[CLS]" => Format::ADDED_CLS,
+      "[SEP]" => Format::ADDED_SEP,
+      "[MASK]" => Format::ADDED_MASK
+    }.freeze
+
+    module_function
+
+    def compile(tokenizer, tokenizer_config = {})
+      profile = audit(tokenizer, tokenizer_config)
+      tokens = vocabulary(tokenizer)
+      validate_added_token_ids(profile.fetch(:added_tokens), tokens)
+      [profile.freeze, tokens]
+    end
+
+    def runtime_meta(profile, tokens)
+      prefix = profile.fetch(:continuing_subword_prefix)
+      {
+        tokenizer_type: Format::TOKENIZER_BERT_WORDPIECE_V1,
+        tokenizer_profile: TOKENIZER_PROFILE,
+        do_lower_case: profile.fetch(:lowercase),
+        strip_accents: profile.fetch(:strip_accents),
+        handle_chinese_chars: profile.fetch(:handle_chinese_chars),
+        clean_text: profile.fetch(:clean_text),
+        added_token_mask: profile.fetch(:added_token_mask),
+        max_input_chars_per_word: profile.fetch(:max_input_chars_per_word),
+        max_token_chars: max_token_chars(tokens, prefix),
+        subword_prefix: prefix,
+        pad_id: token_id(tokens, "[PAD]", 0),
+        unk_id: token_id(tokens, profile.fetch(:unk_token)),
+        cls_id: token_id(tokens, "[CLS]", 0),
+        sep_id: token_id(tokens, "[SEP]", 0),
+        mask_id: token_id(tokens, "[MASK]", 0)
+      }.freeze
+    end
+
+    def audit(tokenizer, tokenizer_config)
+      unsupported!("tokenizer.json is not an object") unless tokenizer.is_a?(Hash)
+      unsupported!("tokenizer_config.json is not an object") unless tokenizer_config.is_a?(Hash)
+      model = tokenizer["model"] || unsupported!("tokenizer.json has no model section")
+      normalizer = tokenizer["normalizer"] || unsupported!("tokenizer has no normalizer")
+      pre_tokenizer = tokenizer["pre_tokenizer"]
+      unsupported!("tokenizer model section is not an object") unless model.is_a?(Hash)
+      unsupported!("tokenizer normalizer is not an object") unless normalizer.is_a?(Hash)
+
+      audit_model(model)
+      audit_normalizer(normalizer)
+      audit_pre_tokenizer(pre_tokenizer)
+      added_tokens = audit_added_tokens(tokenizer["added_tokens"] || [])
+      lowercase = flag(normalizer, "lowercase", tokenizer_config["do_lower_case"], true)
+      clean_text = flag(normalizer, "clean_text", nil, true)
+      max_input_chars = Integer(model.fetch("max_input_chars_per_word", 100))
+      unsupported!("max_input_chars_per_word must be positive") unless max_input_chars.positive?
+      unsupported!("clean_text=false is not supported by the runtime") unless clean_text
+
+      {
+        lowercase: lowercase,
+        strip_accents: strip_accents(normalizer, lowercase),
+        clean_text: clean_text,
+        handle_chinese_chars: flag(normalizer, "handle_chinese_chars",
+                                   tokenizer_config["tokenize_chinese_chars"], true),
+        continuing_subword_prefix: model.fetch("continuing_subword_prefix", "##"),
+        unk_token: model.fetch("unk_token", "[UNK]"),
+        max_input_chars_per_word: max_input_chars,
+        tokenizer_class: tokenizer_config["tokenizer_class"],
+        added_tokens: added_tokens,
+        added_token_mask: added_tokens.reduce(0) do |mask, token|
+          mask | STANDARD_SPECIAL_TOKENS.fetch(token.fetch("content"))
+        end
+      }
+    end
+
+    def audit_model(model)
+      unsupported!("tokenizer model.type is #{model['type'].inspect}, expected WordPiece") unless model["type"] == "WordPiece"
+
+      prefix = model.fetch("continuing_subword_prefix", "##")
+      unsupported!("continuing_subword_prefix #{prefix.inspect} is not supported") unless prefix == "##"
+
+      unk_token = model.fetch("unk_token", "[UNK]")
+      unsupported!("unk_token #{unk_token.inspect} is not supported") unless unk_token == "[UNK]"
+    end
+
+    def audit_normalizer(normalizer)
+      unsupported!("normalizer type #{normalizer['type'].inspect} is not BertNormalizer") unless normalizer["type"] == "BertNormalizer"
+
+      unknown = normalizer.keys - ALLOWED_NORMALIZER_KEYS
+      unsupported!("normalizer has unsupported keys #{unknown.inspect}") unless unknown.empty?
+    end
+
+    def audit_pre_tokenizer(pre_tokenizer)
+      return if pre_tokenizer.is_a?(Hash) && pre_tokenizer["type"] == "BertPreTokenizer"
+
+      unsupported!("pre_tokenizer type #{pre_tokenizer && pre_tokenizer['type'].inspect} is not BertPreTokenizer")
+    end
+
+    def audit_added_tokens(tokens)
+      unsupported!("added_tokens is not an array") unless tokens.is_a?(Array)
+      unsupported!("added_tokens contains a non-object entry") unless tokens.all? { |token| token.is_a?(Hash) }
+      bad = tokens.reject { |token| standard_special?(token) }
+      unsupported!("tokenizer declares non-standard added_tokens #{bad.map { |token| token['content'] }.inspect}") unless bad.empty?
+
+      tokens.each do |token|
+        content = token["content"].to_s
+        unsupported!("added token #{content.inspect} contains whitespace") if content.match?(/\s/)
+        unsupported!("added token #{content.inspect} uses lstrip/rstrip/single_word") if token["lstrip"] || token["rstrip"] || token["single_word"]
+        unsupported!("added token #{content.inspect} must use normalized=false") unless token["normalized"] == false
+      end
+
+      duplicate = tokens.group_by { |token| token["content"] }.find { |_, rows| rows.length > 1 }
+      unsupported!("duplicate added token #{duplicate.first.inspect}") if duplicate
+      tokens.freeze
+    end
+
+    def vocabulary(tokenizer)
+      vocab = tokenizer.dig("model", "vocab") || unsupported!("tokenizer.json has no model.vocab")
+      unsupported!("tokenizer model.vocab is not an object") unless vocab.is_a?(Hash)
+      tokens = Array.new(vocab.length)
+      vocab.each { |token, id| assign_vocab(tokens, token, id) }
+      missing = tokens.index(nil)
+      invalid!("vocab has a hole at id #{missing}") if missing
+      tokens
+    end
+
+    def assign_vocab(tokens, token, id)
+      unsupported!("vocab id #{id.inspect} is not an integer") unless id.is_a?(Integer)
+      invalid!("vocab id #{id} for #{token.inspect} is outside 0...#{tokens.length}") unless id.between?(0, tokens.length - 1)
+      invalid!("duplicate vocab id #{id}") unless tokens[id].nil?
+      tokens[id] = token
+    end
+
+    def validate_added_token_ids(added_tokens, tokens)
+      added_tokens.each do |token|
+        content = token.fetch("content")
+        id = token["id"]
+        unsupported!("added token #{content.inspect} has non-integer id #{id.inspect}") unless id.is_a?(Integer)
+        valid = id.between?(0, tokens.length - 1) && tokens[id] == content
+        unsupported!("added token #{content.inspect} id #{id} does not match model.vocab") unless valid
+      end
+    end
+
+    def flag(hash, key, fallback, default)
+      return !!hash[key] if hash.key?(key) && !hash[key].nil?
+      return !!fallback unless fallback.nil?
+
+      default
+    end
+
+    def strip_accents(normalizer, lowercase)
+      value = normalizer["strip_accents"]
+      normalizer.key?("strip_accents") && !value.nil? ? !!value : lowercase
+    end
+
+    def standard_special?(token)
+      token["special"] && STANDARD_SPECIAL_TOKENS.key?(token["content"])
+    end
+
+    def max_token_chars(tokens, prefix)
+      tokens.reduce(1) do |maximum, token|
+        body = token.start_with?(prefix) ? token[prefix.length..] : token
+        [maximum, body.each_char.count].max
+      end
+    end
+
+    def token_id(tokens, token, fallback = nil)
+      id = tokens.index(token)
+      return id unless id.nil?
+      return fallback unless fallback.nil?
+
+      invalid!("vocabulary has no #{token.inspect}")
+    end
+
+    def unsupported!(message)
+      raise UnsupportedModelError,
+            "#{message}. Supported tokenizer profile: #{TOKENIZER_PROFILE}; " \
+            "other tokenizer behaviour requires a separate runtime capability."
+    end
+
+    def invalid!(message)
+      raise ConversionError, message
+    end
+  end
+end

data/lib/static_embeddings/canonical.rb

@@ -0,0 +1,50 @@
+module StaticEmbeddings
+  module Canonical
+    Dimensions = Struct.new(:native, :output, :trained, keyword_init: true)
+    Runtime = Struct.new(:normalization, :unk_policy, :empty_policy, :max_tokens, :add_special_tokens,
+                         keyword_init: true)
+    Source = Struct.new(:family, :model, :revision, :oracle, :files_sha256, :tokenizer_class,
+                        :config_seq_length, keyword_init: true)
+    Model = Struct.new(:tokens, :matrix, :dimensions, :runtime, :tokenizer, :source, keyword_init: true)
+
+    module_function
+
+    def dimensions(native:, output:, trained: nil)
+      Dimensions.new(native: native, output: output, trained: trained&.freeze).freeze
+    end
+
+    def runtime(normalization:, unk_policy:, empty_policy:, max_tokens:, add_special_tokens: false)
+      Runtime.new(
+        normalization: normalization,
+        unk_policy: unk_policy,
+        empty_policy: empty_policy,
+        max_tokens: max_tokens,
+        add_special_tokens: add_special_tokens
+      ).freeze
+    end
+
+    def source(family:, model:, oracle:, files_sha256:, revision: nil, tokenizer_class: nil,
+               config_seq_length: nil)
+      Source.new(
+        family: family,
+        model: model,
+        revision: revision,
+        oracle: oracle,
+        files_sha256: files_sha256.freeze,
+        tokenizer_class: tokenizer_class,
+        config_seq_length: config_seq_length
+      ).freeze
+    end
+
+    def model(tokens:, matrix:, dimensions:, runtime:, tokenizer:, source:)
+      Model.new(
+        tokens: tokens.freeze,
+        matrix: matrix,
+        dimensions: dimensions,
+        runtime: runtime,
+        tokenizer: tokenizer.freeze,
+        source: source
+      ).freeze
+    end
+  end
+end