@r1z4x/tezgah 1.0.0

package/skills/ai-research/02-tokenization/sentencepiece/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: sentencepiece
+description: Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+tags: [Tokenization, SentencePiece, Language-Independent, BPE, Unigram, Multilingual, CJK Languages, Unicode, Deterministic, Google]
+dependencies: [sentencepiece, transformers]
+---
+
+# SentencePiece - Language-Independent Tokenization
+
+Unsupervised tokenizer that works on raw text without language-specific preprocessing.
+
+## When to use SentencePiece
+
+**Use SentencePiece when:**
+- Building multilingual models (no language-specific rules)
+- Working with CJK languages (Chinese, Japanese, Korean)
+- Need reproducible tokenization (deterministic vocabulary)
+- Want to train on raw text (no pre-tokenization needed)
+- Require lightweight deployment (6MB memory, 50k sentences/sec)
+
+**Performance**:
+- **Speed**: 50,000 sentences/sec
+- **Memory**: ~6MB for loaded model
+- **Languages**: All (language-independent)
+
+**Use alternatives instead**:
+- **HuggingFace Tokenizers**: Faster training, more flexibility
+- **tiktoken**: OpenAI models (GPT-3.5/4)
+- **BERT WordPiece**: English-centric tasks
+
+## Quick start
+
+### Installation
+
+```bash
+# Python
+pip install sentencepiece
+
+# C++ (requires CMake)
+git clone https://github.com/google/sentencepiece.git
+cd sentencepiece
+mkdir build && cd build
+cmake .. && make -j $(nproc)
+sudo make install
+```
+
+### Train model
+
+```bash
+# Command-line (BPE with 8000 vocab)
+spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe
+
+# Python API
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='m',
+    vocab_size=8000,
+    model_type='bpe'
+)
+```
+
+**Training time**: ~1-2 minutes for 100MB corpus
+
+### Encode and decode
+
+```python
+import sentencepiece as spm
+
+# Load model
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Encode to pieces
+pieces = sp.encode('This is a test', out_type=str)
+print(pieces)  # ['▁This', '▁is', '▁a', '▁test']
+
+# Encode to IDs
+ids = sp.encode('This is a test', out_type=int)
+print(ids)  # [284, 47, 11, 1243]
+
+# Decode
+text = sp.decode(ids)
+print(text)  # "This is a test"
+```
+
+## Language-independent design
+
+### Whitespace as symbol (▁)
+
+```python
+text = "Hello world"
+pieces = sp.encode(text, out_type=str)
+print(pieces)  # ['▁Hello', '▁world']
+
+# Decode preserves spaces
+decoded = sp.decode_pieces(pieces)
+print(decoded)  # "Hello world"
+```
+
+**Key principle**: Treat text as raw Unicode, whitespace = ▁ (meta symbol)
+
+## Tokenization algorithms
+
+### BPE (Byte-Pair Encoding)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='bpe_model',
+    vocab_size=16000,
+    model_type='bpe'
+)
+```
+
+**Used by**: mBART
+
+### Unigram (default)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='unigram_model',
+    vocab_size=8000,
+    model_type='unigram'
+)
+```
+
+**Used by**: T5, ALBERT, XLNet
+
+## Training configuration
+
+### Essential parameters
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+    model_type='unigram',
+    character_coverage=0.9995,  # 1.0 for CJK
+    user_defined_symbols=['[SEP]', '[CLS]'],
+    unk_piece='<unk>',
+    num_threads=16
+)
+```
+
+### Character coverage
+
+| Language Type | Coverage | Rationale |
+|---------------|----------|-----------|
+| English       | 0.9995   | Most common chars |
+| CJK (Chinese) | 1.0      | All characters needed |
+| Multilingual  | 0.9995   | Balance |
+
+## Encoding options
+
+### Subword regularization
+
+```python
+# Sample different tokenizations
+for _ in range(3):
+    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)
+    print(pieces)
+
+# Output (different each time):
+# ['▁token', 'ization']
+# ['▁tok', 'en', 'ization']
+```
+
+**Use case**: Data augmentation for robustness.
+
+## Common patterns
+
+### T5-style training
+
+```python
+spm.SentencePieceTrainer.train(
+    input='c4_corpus.txt',
+    model_prefix='t5',
+    vocab_size=32000,
+    model_type='unigram',
+    user_defined_symbols=[f'<extra_id_{i}>' for i in range(100)],
+    unk_id=2,
+    eos_id=1,
+    pad_id=0
+)
+```
+
+### Integration with transformers
+
+```python
+from transformers import T5Tokenizer
+
+# T5 uses SentencePiece internally
+tokenizer = T5Tokenizer.from_pretrained('t5-base')
+inputs = tokenizer('translate English to French: Hello', return_tensors='pt')
+```
+
+## Performance benchmarks
+
+### Training speed
+
+| Corpus | BPE (16k) | Unigram (8k) |
+|--------|-----------|--------------|
+| 100 MB | 1-2 min   | 3-4 min      |
+| 1 GB   | 10-15 min | 30-40 min    |
+
+### Tokenization speed
+
+- **SentencePiece**: 50,000 sentences/sec
+- **HF Tokenizers**: 200,000 sentences/sec (4× faster)
+
+## Supported models
+
+**T5 family**: `t5-base`, `t5-large` (32k vocab, Unigram)
+**ALBERT**: `albert-base-v2` (30k vocab, Unigram)
+**XLNet**: `xlnet-base-cased` (32k vocab, Unigram)
+**mBART**: `facebook/mbart-large-50` (250k vocab, BPE)
+
+## References
+
+- **[Training Guide](references/training.md)** - Detailed options, corpus preparation
+- **[Algorithms](references/algorithms.md)** - BPE vs Unigram, subword regularization
+
+## Resources
+
+- **GitHub**: https://github.com/google/sentencepiece ⭐ 10,000+
+- **Paper**: https://arxiv.org/abs/1808.06226 (EMNLP 2018)
+- **Version**: 0.2.0+
+
+

package/skills/ai-research/02-tokenization/sentencepiece/references/algorithms.md

@@ -0,0 +1,201 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 02-tokenization/sentencepiece/references/algorithms.md; MIT (c) Orchestra Research -->
+# Tokenization Algorithms
+
+BPE vs Unigram comparison and subword regularization.
+
+## BPE (Byte-Pair Encoding)
+
+### Algorithm
+
+1. Initialize vocabulary with characters
+2. Count frequency of adjacent token pairs
+3. Merge most frequent pair
+4. Repeat until vocabulary size reached
+
+### Example
+
+**Corpus**:
+```
+low: 5
+lower: 2
+newest: 6
+widest: 3
+```
+
+**Iteration 1**:
+- Most frequent pair: 'e' + 's' (9 times)
+- Merge → 'es'
+- Vocabulary: [chars] + ['es']
+
+**Iteration 2**:
+- Most frequent: 'es' + 't' (9 times)
+- Merge → 'est'
+- Vocabulary: [chars] + ['es', 'est']
+
+**Result**: `newest` → `new|est`, `widest` → `wid|est`
+
+### Implementation
+
+```python
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='bpe',
+    vocab_size=16000
+)
+```
+
+### Advantages
+
+- Simple algorithm
+- Fast training
+- Good compression ratio
+
+### Disadvantages
+
+- Deterministic (no sampling)
+- May split common words unexpectedly
+
+## Unigram
+
+### Algorithm
+
+1. Start with large vocabulary (all substrings)
+2. Compute probability of each token
+3. Remove tokens with minimal loss impact
+4. Repeat until vocabulary size reached
+
+### Probabilistic tokenization
+
+Given vocabulary with probabilities:
+```
+P('low') = 0.02
+P('est') = 0.03
+P('l') = 0.01
+P('o') = 0.015
+...
+```
+
+Tokenize "lowest":
+```
+Option 1: ['low', 'est']
+P = 0.02 × 0.03 = 0.0006  ← highest
+
+Option 2: ['l', 'o', 'w', 'est']
+P = 0.01 × 0.015 × 0.01 × 0.03 = 0.000000045
+
+Choose option 1 (highest probability)
+```
+
+### Implementation
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='unigram',
+    vocab_size=8000
+)
+```
+
+### Advantages
+
+- Probabilistic (can sample)
+- Better for morphologically rich languages
+- Supports subword regularization
+
+### Disadvantages
+
+- Slower training
+- More complex algorithm
+
+## Comparison
+
+| Feature | BPE | Unigram |
+|---------|-----|---------|
+| Training speed | Fast | Slow |
+| Tokenization | Deterministic | Probabilistic |
+| Sampling | No | Yes |
+| Typical vocab size | 16k-32k | 8k-32k |
+| Used by | mBART | T5, ALBERT, XLNet |
+
+## Subword regularization
+
+Sample different tokenizations during training for robustness.
+
+### Enable sampling
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Sample different tokenizations
+for _ in range(5):
+    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)
+    print(pieces)
+
+# Output (different each time):
+# ['▁token', 'ization']
+# ['▁tok', 'en', 'ization']
+# ['▁token', 'iz', 'ation']
+# ['▁to', 'ken', 'ization']
+# ['▁token', 'ization']
+```
+
+### Parameters
+
+- `alpha`: Regularization strength
+  - 0.0 = deterministic (no sampling)
+  - 0.1 = slight variation
+  - 0.5 = high variation
+  - 1.0 = maximum variation
+
+### Benefits
+
+1. **Robustness**: Model learns multiple tokenizations
+2. **Data augmentation**: More diverse training data
+3. **Better generalization**: Less overfitting to specific tokenization
+
+### Use case
+
+```python
+# Training loop with regularization
+for batch in dataloader:
+    # Sample different tokenizations each epoch
+    tokens = sp.encode(batch['text'], enable_sampling=True, alpha=0.1)
+    # Train model...
+```
+
+**Used by**: mT5, XLM-RoBERTa
+
+## NBest encoding
+
+Get multiple tokenization candidates with scores.
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Get top-5 tokenizations
+nbest = sp.nbest_encode('tokenization', nbest_size=5, out_type=str)
+
+for pieces, score in nbest:
+    print(f"{pieces} (log prob: {score:.4f})")
+
+# Output:
+# ['▁token', 'ization'] (log prob: -2.34)
+# ['▁tok', 'en', 'ization'] (log prob: -2.41)
+# ['▁token', 'iz', 'ation'] (log prob: -2.57)
+```
+
+### Use cases
+
+1. **Ensemble tokenization**: Average over multiple tokenizations
+2. **Uncertainty estimation**: Check variance in scores
+3. **Debugging**: Understand tokenizer behavior
+
+## Best practices
+
+1. **Use Unigram for multilingual** - Better for diverse languages
+2. **Use BPE for speed** - Faster training and inference
+3. **Enable subword regularization** - Improves model robustness
+4. **Set alpha=0.1 for slight variation** - Good balance
+5. **Use deterministic mode for inference** - Consistent results