primoji 0.1.0__tar.gz

primoji-0.1.0/PKG-INFO

@@ -0,0 +1,256 @@
+Metadata-Version: 2.4
+Name: primoji
+Version: 0.1.0
+Summary: A compositional semantic tokenizer for LLMs based on NSM primitives, emoji, and common word tokens
+Author-email: Frane Bandov <frane@offbyte.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/frane/primoji
+Project-URL: Documentation, https://github.com/frane/primoji#readme
+Project-URL: Repository, https://github.com/frane/primoji
+Project-URL: Issues, https://github.com/frane/primoji/issues
+Keywords: tokenizer,tokenization,nlp,llm,language-model,semantic,compositional,emoji,nsm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: emoji>=2.14
+Provides-Extra: dev
+Requires-Dist: pytest>=8.3; extra == "dev"
+Requires-Dist: black>=24.10; extra == "dev"
+Requires-Dist: ruff>=0.8; extra == "dev"
+Provides-Extra: build-dict
+Requires-Dist: spacy>=3.8; extra == "build-dict"
+Requires-Dist: wordfreq>=3.0; extra == "build-dict"
+Provides-Extra: train
+Requires-Dist: torch>=2.5; extra == "train"
+Requires-Dist: tokenizers>=0.21; extra == "train"
+Requires-Dist: datasets>=3.0; extra == "train"
+Requires-Dist: numpy>=1.26; extra == "train"
+Provides-Extra: spelling
+Requires-Dist: symspellpy>=6.7; extra == "spelling"
+Provides-Extra: all
+Requires-Dist: primoji[build-dict,dev,spelling,train]; extra == "all"
+
+# Primoji
+
+A compositional semantic tokenizer for language models, built from Natural
+Semantic Metalanguage (NSM) primitives, Unicode emoji, common word tokens,
+and a UTF-8 byte fallback.
+
+```python
+from primoji import Tokenizer
+
+tok = Tokenizer()
+ids = tok.encode("The teacher explained photosynthesis")
+tok.decode(ids)
+# 'teacher say photosynthesis'
+
+tok.vocab_size
+# 10195
+```
+
+## Install
+
+```bash
+pip install primoji
+```
+
+Requires Python 3.10+. Only runtime dependency is `emoji`. Optional extras
+for the dictionary build pipeline (`spacy`, `wordfreq`), training scripts
+(`torch`, `tokenizers`, `datasets`), or fuzzy matching (`symspellpy`):
+
+```bash
+pip install "primoji[build-dict]"   # to rebuild the dictionary from sources
+pip install "primoji[train]"        # to run the training scripts
+pip install "primoji[spelling]"     # to enable conservative SymSpell fuzzy matching
+pip install "primoji[all]"          # everything (also includes dev tools)
+```
+
+## What it is
+
+Primoji is a tokenizer that constructs token meanings compositionally. Instead
+of discovering subword units from corpus statistics like BPE, it ships with a
+fixed compositional vocabulary of about 10,000 tokens: 1,175 Unicode emoji,
+140 NSM semantic primitives, ~7,700 common word tokens, ~830 structural and
+geographic tokens (flags, anchors, punctuation, math), and 258 byte-fallback
+tokens. Rare and technical words decompose into short sequences of primitives
+(`[PLANT, HAVE, LIGHT]` for photosynthesis, `[WATER, CAUSE, AIR]` for
+evaporation). The encode pipeline never produces an UNK token.
+
+When you might want it: research on tokenization, controlled experiments
+that need a vocabulary 3x smaller than BPE at near-parity compression,
+multilingual experiments where the primitive set transfers across languages
+by design, small-vocab architectures whose cost scales with vocabulary size,
+or educational and scientific text where compositional technical vocabulary
+is dense. It is not a drop-in BPE replacement for production English chatbots:
+generation is lossy at the word level (decoded text reconstructs concepts but
+not always exact surface forms), and creative or stylistic writing loses
+distinctions like "melancholy" vs "sadness" that BPE preserves as distinct
+tokens.
+
+## Quick start
+
+```python
+from primoji import Tokenizer
+
+tok = Tokenizer()
+
+# Encode and decode
+ids = tok.encode("Water evaporates when heated")
+print(ids)
+# [1266, ..., ...]
+print(tok.decode(ids))
+
+# Inspect a token
+print(tok.describe(ids[0]))
+# '💧 (ID 1266) — Tier 2 primitive: WATER — Liquid, fluid'
+
+# Classify a word by which tier handles it
+print(tok.classify_word("photosynthesis"))   # 'dict_composed'
+print(tok.classify_word("water"))            # 'tier2_primitive'
+print(tok.classify_word("the"))              # 'dict_dropped'
+print(tok.classify_word("Mediterranean"))    # 'tier3_anchor' or 'byte_fallback'
+
+# Vocabulary size
+print(tok.vocab_size)                        # 10195
+```
+
+## Vocabulary structure
+
+Four tiers with strict encoding precedence. The model only ever sees integer
+IDs; the tier labels exist for diagnostics and the dictionary build.
+
+| Tier | Count | Examples |
+|------|------:|----------|
+| 1a Direct emoji              | ~1,175 | 🐕 (dog), 🏠 (house), ☀️ (sun) |
+| 1b Common word tokens        | ~7,700 | government, important, however |
+| 2  Compositional primitives  |    140 | PLANT, HAVE, LIGHT, MOVE, KNOW |
+| 3  Structural and geographic |   ~830 | 🇩🇪, 🇫🇷, punctuation, digits, math operators, NER anchors |
+| 4  UTF-8 byte fallback       |    258 | 256 byte tokens + 2 boundary markers |
+
+The 140 primitives are 65 canonical NSM primes from
+Wierzbicka's Natural Semantic Metalanguage plus 75 domain expansions
+(perceptual, grammatical, scientific, social). Country flags act as geographic
+modifiers in compositions. Anchors are the top ~500 proper nouns extracted
+from FineWeb-Edu via spaCy NER.
+
+A note on emoji: the model never sees glyphs, only integer IDs. Tier 1a is
+just a convenient source of ~1,200 visually distinct token slots for concrete
+nouns. In generated output, emoji tokens decode to their English words
+("dog", "house", "sun"), not to Unicode emoji.
+
+## Configuration
+
+The dictionary, primitive set, and word list are all swappable. Five tunable
+dimensions:
+
+1. **Vocabulary size.** From ~5,300 tokens (~6% composed text) to ~18,000
+   tokens (~2% composed). Controls the trade-off between compression and
+   semantic-structure signal.
+2. **Composition depth.** Maximum number of primitives per concept. Default 5.
+3. **Primitive count.** Add domain-specific primitives (e.g.\ HEALTH for
+   medical text) without touching the rest of the system.
+4. **Composition rate.** A consequence of the above three.
+5. **Domain specialization.** A medical Primoji can have 3,000 medical
+   compositions on top of 10,000 general words. The dictionary build pipeline
+   accepts a target corpus and produces a corresponding word list.
+
+Rebuild the dictionary from sources:
+
+```bash
+pip install "primoji[build-dict]"
+python -m scripts.build_dictionary
+```
+
+This regenerates `data/dictionary_seed.json` from layered sources: Unicode
+CLDR annotations, NSM primitive synonyms, NER anchors, WordNet
+auto-compositions. Layer precedence is fixed: primitive synonyms override
+emoji catalog entries, single-primitive word tokens override compositions to
+preserve reverse lookup, and ELCo and emoji2vec mappings supplement CLDR base
+entries.
+
+## Training your own model
+
+Reference training scripts for 125M and 1B GPT-style models live in
+`scripts/`:
+
+```bash
+# Tokenize FineWeb-Edu shards (or any HuggingFace dataset)
+python -m scripts.prepare_training_data --n-docs 500000
+
+# Train 125M
+python -m scripts.train --tokenizer primoji --v2 --byte-weight 0.7
+
+# Train 1B with gradient accumulation
+python -m scripts.train --tokenizer primoji --model-size 1b \
+  --batch-size 4 --grad-accum 8 --v2 --byte-weight 0.7
+
+# Train BPE baseline on the same data
+python -m scripts.train --tokenizer mistral
+```
+
+Bits-per-byte (BPB) is computed correctly across vocabulary sizes for fair
+cross-tokenizer comparison. See `scripts/train.py` for the full
+hyperparameter list.
+
+## Limitations
+
+- English only. The NSM primes are verified across 30+ language families,
+  but the dictionary, word list, and evaluation are all English. Multilingual
+  evaluation is future work.
+- Lossy at the word level. Known compositions decode exactly; novel
+  compositions decode to their primitive names, which is sufficient for
+  training-efficiency studies (BPB) but not for verbatim text generation
+  without an auxiliary surface-form mechanism.
+- Designed for educational and scientific text. Compression and learning
+  benefits are strongest where vocabulary is compositional and
+  concept-dense. Conversational, creative, or code-heavy text will not see
+  the same gains.
+- Not yet tested at scale beyond 1B parameters.
+
+## Tests
+
+```bash
+pip install "primoji[dev]"
+pytest
+```
+
+616 tests covering encode/decode round-trips, byte-fallback coverage,
+compositional embedding correctness, ID-range invariants, and frozen
+vocabulary boundaries.
+
+## Paper
+
+The full paper is included in this repository as
+[`paper.pdf`](paper.pdf). It is not yet on arXiv. If you reference this work,
+please cite as:
+
+```bibtex
+@unpublished{bandov2026primoji,
+  title  = {Primoji: Compositional Semantic Tokenization for Language Model Training},
+  author = {Bandov, Frane},
+  year   = {2026},
+  note   = {Manuscript. \url{https://github.com/frane/primoji/blob/master/paper.pdf}},
+}
+```
+
+## License
+
+Apache 2.0. See `LICENSE`.
+
+## Acknowledgments
+
+Anna Wierzbicka and the wider Natural Semantic Metalanguage research community
+for the linguistic foundation. The FineWeb-Edu team at HuggingFace for the
+training corpus. The Unicode CLDR project for emoji annotations.

primoji-0.1.0/README.md

@@ -0,0 +1,212 @@
+# Primoji
+
+A compositional semantic tokenizer for language models, built from Natural
+Semantic Metalanguage (NSM) primitives, Unicode emoji, common word tokens,
+and a UTF-8 byte fallback.
+
+```python
+from primoji import Tokenizer
+
+tok = Tokenizer()
+ids = tok.encode("The teacher explained photosynthesis")
+tok.decode(ids)
+# 'teacher say photosynthesis'
+
+tok.vocab_size
+# 10195
+```
+
+## Install
+
+```bash
+pip install primoji
+```
+
+Requires Python 3.10+. Only runtime dependency is `emoji`. Optional extras
+for the dictionary build pipeline (`spacy`, `wordfreq`), training scripts
+(`torch`, `tokenizers`, `datasets`), or fuzzy matching (`symspellpy`):
+
+```bash
+pip install "primoji[build-dict]"   # to rebuild the dictionary from sources
+pip install "primoji[train]"        # to run the training scripts
+pip install "primoji[spelling]"     # to enable conservative SymSpell fuzzy matching
+pip install "primoji[all]"          # everything (also includes dev tools)
+```
+
+## What it is
+
+Primoji is a tokenizer that constructs token meanings compositionally. Instead
+of discovering subword units from corpus statistics like BPE, it ships with a
+fixed compositional vocabulary of about 10,000 tokens: 1,175 Unicode emoji,
+140 NSM semantic primitives, ~7,700 common word tokens, ~830 structural and
+geographic tokens (flags, anchors, punctuation, math), and 258 byte-fallback
+tokens. Rare and technical words decompose into short sequences of primitives
+(`[PLANT, HAVE, LIGHT]` for photosynthesis, `[WATER, CAUSE, AIR]` for
+evaporation). The encode pipeline never produces an UNK token.
+
+When you might want it: research on tokenization, controlled experiments
+that need a vocabulary 3x smaller than BPE at near-parity compression,
+multilingual experiments where the primitive set transfers across languages
+by design, small-vocab architectures whose cost scales with vocabulary size,
+or educational and scientific text where compositional technical vocabulary
+is dense. It is not a drop-in BPE replacement for production English chatbots:
+generation is lossy at the word level (decoded text reconstructs concepts but
+not always exact surface forms), and creative or stylistic writing loses
+distinctions like "melancholy" vs "sadness" that BPE preserves as distinct
+tokens.
+
+## Quick start
+
+```python
+from primoji import Tokenizer
+
+tok = Tokenizer()
+
+# Encode and decode
+ids = tok.encode("Water evaporates when heated")
+print(ids)
+# [1266, ..., ...]
+print(tok.decode(ids))
+
+# Inspect a token
+print(tok.describe(ids[0]))
+# '💧 (ID 1266) — Tier 2 primitive: WATER — Liquid, fluid'
+
+# Classify a word by which tier handles it
+print(tok.classify_word("photosynthesis"))   # 'dict_composed'
+print(tok.classify_word("water"))            # 'tier2_primitive'
+print(tok.classify_word("the"))              # 'dict_dropped'
+print(tok.classify_word("Mediterranean"))    # 'tier3_anchor' or 'byte_fallback'
+
+# Vocabulary size
+print(tok.vocab_size)                        # 10195
+```
+
+## Vocabulary structure
+
+Four tiers with strict encoding precedence. The model only ever sees integer
+IDs; the tier labels exist for diagnostics and the dictionary build.
+
+| Tier | Count | Examples |
+|------|------:|----------|
+| 1a Direct emoji              | ~1,175 | 🐕 (dog), 🏠 (house), ☀️ (sun) |
+| 1b Common word tokens        | ~7,700 | government, important, however |
+| 2  Compositional primitives  |    140 | PLANT, HAVE, LIGHT, MOVE, KNOW |
+| 3  Structural and geographic |   ~830 | 🇩🇪, 🇫🇷, punctuation, digits, math operators, NER anchors |
+| 4  UTF-8 byte fallback       |    258 | 256 byte tokens + 2 boundary markers |
+
+The 140 primitives are 65 canonical NSM primes from
+Wierzbicka's Natural Semantic Metalanguage plus 75 domain expansions
+(perceptual, grammatical, scientific, social). Country flags act as geographic
+modifiers in compositions. Anchors are the top ~500 proper nouns extracted
+from FineWeb-Edu via spaCy NER.
+
+A note on emoji: the model never sees glyphs, only integer IDs. Tier 1a is
+just a convenient source of ~1,200 visually distinct token slots for concrete
+nouns. In generated output, emoji tokens decode to their English words
+("dog", "house", "sun"), not to Unicode emoji.
+
+## Configuration
+
+The dictionary, primitive set, and word list are all swappable. Five tunable
+dimensions:
+
+1. **Vocabulary size.** From ~5,300 tokens (~6% composed text) to ~18,000
+   tokens (~2% composed). Controls the trade-off between compression and
+   semantic-structure signal.
+2. **Composition depth.** Maximum number of primitives per concept. Default 5.
+3. **Primitive count.** Add domain-specific primitives (e.g.\ HEALTH for
+   medical text) without touching the rest of the system.
+4. **Composition rate.** A consequence of the above three.
+5. **Domain specialization.** A medical Primoji can have 3,000 medical
+   compositions on top of 10,000 general words. The dictionary build pipeline
+   accepts a target corpus and produces a corresponding word list.
+
+Rebuild the dictionary from sources:
+
+```bash
+pip install "primoji[build-dict]"
+python -m scripts.build_dictionary
+```
+
+This regenerates `data/dictionary_seed.json` from layered sources: Unicode
+CLDR annotations, NSM primitive synonyms, NER anchors, WordNet
+auto-compositions. Layer precedence is fixed: primitive synonyms override
+emoji catalog entries, single-primitive word tokens override compositions to
+preserve reverse lookup, and ELCo and emoji2vec mappings supplement CLDR base
+entries.
+
+## Training your own model
+
+Reference training scripts for 125M and 1B GPT-style models live in
+`scripts/`:
+
+```bash
+# Tokenize FineWeb-Edu shards (or any HuggingFace dataset)
+python -m scripts.prepare_training_data --n-docs 500000
+
+# Train 125M
+python -m scripts.train --tokenizer primoji --v2 --byte-weight 0.7
+
+# Train 1B with gradient accumulation
+python -m scripts.train --tokenizer primoji --model-size 1b \
+  --batch-size 4 --grad-accum 8 --v2 --byte-weight 0.7
+
+# Train BPE baseline on the same data
+python -m scripts.train --tokenizer mistral
+```
+
+Bits-per-byte (BPB) is computed correctly across vocabulary sizes for fair
+cross-tokenizer comparison. See `scripts/train.py` for the full
+hyperparameter list.
+
+## Limitations
+
+- English only. The NSM primes are verified across 30+ language families,
+  but the dictionary, word list, and evaluation are all English. Multilingual
+  evaluation is future work.
+- Lossy at the word level. Known compositions decode exactly; novel
+  compositions decode to their primitive names, which is sufficient for
+  training-efficiency studies (BPB) but not for verbatim text generation
+  without an auxiliary surface-form mechanism.
+- Designed for educational and scientific text. Compression and learning
+  benefits are strongest where vocabulary is compositional and
+  concept-dense. Conversational, creative, or code-heavy text will not see
+  the same gains.
+- Not yet tested at scale beyond 1B parameters.
+
+## Tests
+
+```bash
+pip install "primoji[dev]"
+pytest
+```
+
+616 tests covering encode/decode round-trips, byte-fallback coverage,
+compositional embedding correctness, ID-range invariants, and frozen
+vocabulary boundaries.
+
+## Paper
+
+The full paper is included in this repository as
+[`paper.pdf`](paper.pdf). It is not yet on arXiv. If you reference this work,
+please cite as:
+
+```bibtex
+@unpublished{bandov2026primoji,
+  title  = {Primoji: Compositional Semantic Tokenization for Language Model Training},
+  author = {Bandov, Frane},
+  year   = {2026},
+  note   = {Manuscript. \url{https://github.com/frane/primoji/blob/master/paper.pdf}},
+}
+```
+
+## License
+
+Apache 2.0. See `LICENSE`.
+
+## Acknowledgments
+
+Anna Wierzbicka and the wider Natural Semantic Metalanguage research community
+for the linguistic foundation. The FineWeb-Edu team at HuggingFace for the
+training corpus. The Unicode CLDR project for emoji annotations.

primoji-0.1.0/primoji/__init__.py

@@ -0,0 +1,22 @@
+"""Primoji: a compositional semantic tokenizer for LLMs.
+
+Encodes English text into a vocabulary of roughly 10,000 tokens built from
+Unicode emoji, Natural Semantic Metalanguage (NSM) primitives, common word
+tokens, and a UTF-8 byte fallback. The encode pipeline never produces an
+UNK token: any input is representable.
+
+Example:
+    >>> from primoji import Tokenizer
+    >>> tok = Tokenizer()
+    >>> ids = tok.encode("The teacher explained photosynthesis")
+    >>> tok.decode(ids)
+    'teacher say photosynthesis'
+    >>> tok.vocab_size
+    10195
+"""
+
+from primoji.byte_fallback import decode_bytes, encode_bytes
+from primoji.tokenizer import Tokenizer
+
+__all__ = ["Tokenizer", "encode_bytes", "decode_bytes"]
+__version__ = "0.1.0"

primoji-0.1.0/primoji/alias_map.py

@@ -0,0 +1,104 @@
+"""Grammar word alias map for compositional embeddings.
+
+Maps grammar word token IDs to their primitive component IDs.
+"is" (word token) -> [BE primitive ID, NOW primitive ID]
+"was" (word token) -> [BE primitive ID, BEFORE primitive ID]
+
+The model uses these to compute grammar word embeddings as the mean
+of their primitive component embeddings. This gives grammar words
+semantic structure without consuming them as primitives in running text.
+"""
+
+from __future__ import annotations
+
+from primoji.primitives import get_primitive_by_name
+
+# Grammar word -> primitive decomposition (by name)
+GRAMMAR_ALIASES: dict[str, list[str]] = {
+    # Copula/be verbs
+    "is": ["BE", "NOW"], "are": ["BE", "NOW"], "am": ["BE", "NOW"],
+    "was": ["BE", "BEFORE"], "were": ["BE", "BEFORE"],
+    "be": ["BE"], "been": ["BE", "BEFORE"], "being": ["BE"],
+
+    # Have verbs
+    "has": ["HAVE", "NOW"], "have": ["HAVE"], "had": ["HAVE", "BEFORE"],
+
+    # Do verbs
+    "do": ["DO"], "does": ["DO", "NOW"], "did": ["DO", "BEFORE"],
+
+    # Modals
+    "can": ["CAN"], "could": ["CAN", "BEFORE"],
+    "will": ["AFTER"], "would": ["WANT", "BEFORE"],
+    "should": ["GOOD", "DO"], "may": ["MAYBE"],
+    "might": ["MAYBE", "BEFORE"], "must": ["WANT", "VERY"],
+    "shall": ["AFTER"],
+
+    # Negation
+    "not": ["NOT"], "no": ["NOT"], "never": ["NOT", "TIME"],
+
+    # Pronouns
+    "i": ["SOMEONE", "THIS"], "me": ["SOMEONE", "THIS"],
+    "my": ["SOMEONE", "THIS"],
+    "you": ["SOMEONE", "OTHER"], "your": ["SOMEONE", "OTHER"],
+    "he": ["SOMEONE"], "him": ["SOMEONE"], "his": ["SOMEONE"],
+    "she": ["SOMEONE"], "her": ["SOMEONE"],
+    "it": ["SOMETHING"], "its": ["SOMETHING"],
+    "we": ["SOMEONE", "THIS", "MANY"], "us": ["SOMEONE", "THIS", "MANY"],
+    "our": ["SOMEONE", "THIS", "MANY"],
+    "they": ["SOMEONE", "OTHER", "MANY"], "them": ["SOMEONE", "OTHER", "MANY"],
+    "their": ["SOMEONE", "OTHER", "MANY"],
+
+    # Determiners
+    "this": ["THIS"], "that": ["OTHER"],
+    "these": ["THIS", "MANY"], "those": ["OTHER", "MANY"],
+    "all": ["ALL"], "every": ["ALL"],
+    "some": ["SOME"], "each": ["ALL", "ONE"],
+    "any": ["SOME"], "many": ["MANY"], "few": ["FEW"],
+    "much": ["BIG"], "more": ["MORE"], "most": ["MANY", "VERY"],
+
+    # Prepositions
+    "with": ["WITH"], "for": ["FOR"], "about": ["ABOUT"],
+    "above": ["ABOVE"], "below": ["BELOW"],
+    "near": ["NEAR"], "far": ["FAR"],
+    "before": ["BEFORE"], "after": ["AFTER"],
+    "here": ["HERE"], "there": ["THERE_IS"], "where": ["WHERE"],
+
+    # Conjunctions/logic
+    "if": ["IF"], "because": ["BECAUSE"],
+    "like": ["LIKE_AS"], "as": ["LIKE_AS"],
+
+    # Adverbs
+    "very": ["VERY"], "now": ["NOW"],
+    "also": ["ADD"],
+    "always": ["ALL", "TIME"], "sometimes": ["SOME", "TIME"],
+    "often": ["MANY", "TIME"], "usually": ["MANY", "TIME"],
+}
+
+
+def build_alias_map(encode_fn: callable) -> dict[int, list[int]]:
+    """Convert GRAMMAR_ALIASES to token ID -> primitive ID list.
+
+    Args:
+        encode_fn: function that takes a word and returns token IDs.
+
+    Returns:
+        Dict mapping word token ID -> list of primitive token IDs.
+    """
+    alias_map: dict[int, list[int]] = {}
+
+    for word, prim_names in GRAMMAR_ALIASES.items():
+        word_ids = encode_fn(word)
+        if len(word_ids) != 1:
+            continue  # skip if word doesn't encode to single token
+
+        tok_id = word_ids[0]
+        prim_ids = []
+        for pname in prim_names:
+            p = get_primitive_by_name(pname)
+            if p is not None:
+                prim_ids.append(p.id)
+
+        if prim_ids:
+            alias_map[tok_id] = prim_ids
+
+    return alias_map

primoji-0.1.0/primoji/byte_fallback.py

@@ -0,0 +1,80 @@
+"""Byte fallback encoder for unknown words.
+
+When a word isn't found in the emoji dictionary and SymSpell can't correct it,
+encode it as raw UTF-8 bytes wrapped in boundary markers. This guarantees
+zero UNK tokens and zero information loss.
+
+Follows the same pattern as SentencePiece's byte_fallback (used by Llama 1/2/3).
+"""
+
+from __future__ import annotations
+
+from primoji.utils import _IDS
+
+# Token IDs for byte fallback (dynamically computed from data sizes)
+BYTES_START_ID: int = _IDS["BYTES_START"]
+BYTES_END_ID: int = _IDS["BYTES_END"]
+BYTE_TOKEN_OFFSET: int = _IDS["BYTE_OFFSET"]
+
+
+def encode_bytes(word: str) -> list[int]:
+    """Encode a word as UTF-8 byte token IDs with boundary markers.
+
+    Args:
+        word: Unknown word to encode as bytes.
+
+    Returns:
+        List of token IDs: [BYTES_START] + byte_ids + [BYTES_END]
+    """
+    byte_ids = [BYTE_TOKEN_OFFSET + b for b in word.encode("utf-8")]
+    return [BYTES_START_ID] + byte_ids + [BYTES_END_ID]
+
+
+def decode_bytes(ids: list[int]) -> str:
+    """Decode byte token IDs back to a string.
+
+    Args:
+        ids: Token ID sequence starting with BYTES_START, ending with BYTES_END.
+
+    Returns:
+        Decoded string.
+
+    Raises:
+        ValueError: If markers are missing or bytes are invalid UTF-8.
+    """
+    if not ids:
+        raise ValueError("Empty token ID sequence")
+    if ids[0] != BYTES_START_ID:
+        raise ValueError(f"Expected BYTES_START ({BYTES_START_ID}), got {ids[0]}")
+
+    end_idx = -1
+    for i in range(1, len(ids)):
+        if ids[i] == BYTES_END_ID:
+            end_idx = i
+            break
+    if end_idx == -1:
+        raise ValueError(f"Missing BYTES_END ({BYTES_END_ID}) marker")
+
+    byte_values = []
+    for tid in ids[1:end_idx]:
+        bval = tid - BYTE_TOKEN_OFFSET
+        if not (0 <= bval <= 255):
+            raise ValueError(f"Token ID {tid} is not a valid byte token")
+        byte_values.append(bval)
+
+    return bytes(byte_values).decode("utf-8")
+
+
+def is_byte_token(token_id: int) -> bool:
+    """Check if a token ID is a byte fallback token (0x00-0xFF range)."""
+    return BYTE_TOKEN_OFFSET <= token_id <= BYTE_TOKEN_OFFSET + 255
+
+
+def is_byte_boundary(token_id: int) -> bool:
+    """Check if a token ID is a byte boundary marker (START or END)."""
+    return token_id in (BYTES_START_ID, BYTES_END_ID)
+
+
+def is_byte_region_token(token_id: int) -> bool:
+    """Check if a token ID is any byte-related token (boundary or data)."""
+    return is_byte_boundary(token_id) or is_byte_token(token_id)