@synsci/cli-darwin-x64 1.1.49

package/bin/skills/sentencepiece/references/algorithms.md

@@ -0,0 +1,200 @@
+# Tokenization Algorithms
+
+BPE vs Unigram comparison and subword regularization.
+
+## BPE (Byte-Pair Encoding)
+
+### Algorithm
+
+1. Initialize vocabulary with characters
+2. Count frequency of adjacent token pairs
+3. Merge most frequent pair
+4. Repeat until vocabulary size reached
+
+### Example
+
+**Corpus**:
+```
+low: 5
+lower: 2
+newest: 6
+widest: 3
+```
+
+**Iteration 1**:
+- Most frequent pair: 'e' + 's' (9 times)
+- Merge → 'es'
+- Vocabulary: [chars] + ['es']
+
+**Iteration 2**:
+- Most frequent: 'es' + 't' (9 times)
+- Merge → 'est'
+- Vocabulary: [chars] + ['es', 'est']
+
+**Result**: `newest` → `new|est`, `widest` → `wid|est`
+
+### Implementation
+
+```python
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='bpe',
+    vocab_size=16000
+)
+```
+
+### Advantages
+
+- Simple algorithm
+- Fast training
+- Good compression ratio
+
+### Disadvantages
+
+- Deterministic (no sampling)
+- May split common words unexpectedly
+
+## Unigram
+
+### Algorithm
+
+1. Start with large vocabulary (all substrings)
+2. Compute probability of each token
+3. Remove tokens with minimal loss impact
+4. Repeat until vocabulary size reached
+
+### Probabilistic tokenization
+
+Given vocabulary with probabilities:
+```
+P('low') = 0.02
+P('est') = 0.03
+P('l') = 0.01
+P('o') = 0.015
+...
+```
+
+Tokenize "lowest":
+```
+Option 1: ['low', 'est']
+P = 0.02 × 0.03 = 0.0006  ← highest
+
+Option 2: ['l', 'o', 'w', 'est']
+P = 0.01 × 0.015 × 0.01 × 0.03 = 0.000000045
+
+Choose option 1 (highest probability)
+```
+
+### Implementation
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='unigram',
+    vocab_size=8000
+)
+```
+
+### Advantages
+
+- Probabilistic (can sample)
+- Better for morphologically rich languages
+- Supports subword regularization
+
+### Disadvantages
+
+- Slower training
+- More complex algorithm
+
+## Comparison
+
+| Feature | BPE | Unigram |
+|---------|-----|---------|
+| Training speed | Fast | Slow |
+| Tokenization | Deterministic | Probabilistic |
+| Sampling | No | Yes |
+| Typical vocab size | 16k-32k | 8k-32k |
+| Used by | mBART | T5, ALBERT, XLNet |
+
+## Subword regularization
+
+Sample different tokenizations during training for robustness.
+
+### Enable sampling
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Sample different tokenizations
+for _ in range(5):
+    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)
+    print(pieces)
+
+# Output (different each time):
+# ['▁token', 'ization']
+# ['▁tok', 'en', 'ization']
+# ['▁token', 'iz', 'ation']
+# ['▁to', 'ken', 'ization']
+# ['▁token', 'ization']
+```
+
+### Parameters
+
+- `alpha`: Regularization strength
+  - 0.0 = deterministic (no sampling)
+  - 0.1 = slight variation
+  - 0.5 = high variation
+  - 1.0 = maximum variation
+
+### Benefits
+
+1. **Robustness**: Model learns multiple tokenizations
+2. **Data augmentation**: More diverse training data
+3. **Better generalization**: Less overfitting to specific tokenization
+
+### Use case
+
+```python
+# Training loop with regularization
+for batch in dataloader:
+    # Sample different tokenizations each epoch
+    tokens = sp.encode(batch['text'], enable_sampling=True, alpha=0.1)
+    # Train model...
+```
+
+**Used by**: mT5, XLM-RoBERTa
+
+## NBest encoding
+
+Get multiple tokenization candidates with scores.
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Get top-5 tokenizations
+nbest = sp.nbest_encode('tokenization', nbest_size=5, out_type=str)
+
+for pieces, score in nbest:
+    print(f"{pieces} (log prob: {score:.4f})")
+
+# Output:
+# ['▁token', 'ization'] (log prob: -2.34)
+# ['▁tok', 'en', 'ization'] (log prob: -2.41)
+# ['▁token', 'iz', 'ation'] (log prob: -2.57)
+```
+
+### Use cases
+
+1. **Ensemble tokenization**: Average over multiple tokenizations
+2. **Uncertainty estimation**: Check variance in scores
+3. **Debugging**: Understand tokenizer behavior
+
+## Best practices
+
+1. **Use Unigram for multilingual** - Better for diverse languages
+2. **Use BPE for speed** - Faster training and inference
+3. **Enable subword regularization** - Improves model robustness
+4. **Set alpha=0.1 for slight variation** - Good balance
+5. **Use deterministic mode for inference** - Consistent results

package/bin/skills/sentencepiece/references/training.md

@@ -0,0 +1,304 @@
+# SentencePiece Training Guide
+
+Complete guide to training SentencePiece models.
+
+## Training workflow
+
+### Step 1: Prepare corpus
+
+```bash
+# Plain text file, one sentence per line (recommended)
+cat corpus.txt
+# Hello world
+# This is a test
+# SentencePiece is language-independent
+
+# Or use raw text (SentencePiece handles sentence splitting)
+```
+
+### Step 2: Train model
+
+**Command-line**:
+```bash
+spm_train \
+  --input=corpus.txt \
+  --model_prefix=m \
+  --vocab_size=8000 \
+  --model_type=unigram \
+  --character_coverage=0.9995
+```
+
+**Python API**:
+```python
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=8000,
+    model_type='unigram'
+)
+```
+
+**Output**: `m.model` (binary), `m.vocab` (text vocabulary)
+
+### Step 3: Load and use
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+pieces = sp.encode('Test sentence', out_type=str)
+```
+
+## Training parameters
+
+### Core parameters
+
+```python
+spm.SentencePieceTrainer.train(
+    # Required
+    input='corpus.txt',           # Input corpus
+    model_prefix='output',        # Output prefix
+    vocab_size=8000,              # Target vocabulary size
+
+    # Algorithm
+    model_type='unigram',         # 'unigram', 'bpe', 'char', 'word'
+
+    # Coverage
+    character_coverage=0.9995,    # 0.9995 for most, 1.0 for CJK
+
+    # Normalization
+    normalization_rule_name='nmt_nfkc',  # 'nmt_nfkc', 'nfkc', 'identity'
+
+    # Performance
+    num_threads=16,               # Training threads
+    input_sentence_size=10000000  # Max sentences to load
+)
+```
+
+### Special tokens
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+
+    # Control symbols (special tokens for model control)
+    control_symbols=['<s>', '</s>', '<pad>'],
+
+    # User-defined symbols (never split)
+    user_defined_symbols=['[MASK]', '[SEP]', '[CLS]'],
+
+    # Special token pieces
+    unk_piece='<unk>',
+    bos_piece='<s>',
+    eos_piece='</s>',
+    pad_piece='<pad>',
+
+    # Special token IDs
+    unk_id=0,
+    bos_id=1,
+    eos_id=2,
+    pad_id=3
+)
+```
+
+### Advanced options
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+
+    # Byte fallback (handle unknown chars)
+    byte_fallback=True,
+
+    # Digit handling
+    split_digits=True,            # Split digits individually
+
+    # Script splitting
+    split_by_unicode_script=True, # Split by Unicode script
+    split_by_whitespace=True,     # Split by whitespace
+
+    # Length constraints
+    max_sentencepiece_length=16,  # Max token length
+
+    # Rare word handling
+    min_frequency=2,              # Min frequency for token
+
+    # Training size
+    input_sentence_size=10000000, # Max sentences
+    shuffle_input_sentence=True,  # Shuffle training data
+
+    # Seed
+    seed_sentencepiece_size=1000000  # Seed vocab size
+)
+```
+
+## Training from Python iterator
+
+```python
+import sentencepiece as spm
+from datasets import load_dataset
+
+# Load dataset
+dataset = load_dataset('wikitext', 'wikitext-103-raw-v1', split='train')
+
+# Create iterator
+def corpus_iterator():
+    for example in dataset:
+        if example['text'].strip():
+            yield example['text']
+
+# Train from iterator
+spm.SentencePieceTrainer.train(
+    sentence_iterator=corpus_iterator(),
+    model_prefix='wiki',
+    vocab_size=32000,
+    model_type='unigram'
+)
+```
+
+## Model types
+
+### BPE
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='bpe',
+    vocab_size=16000
+)
+```
+
+**Training time**: ~10-15 min for 1GB corpus
+
+### Unigram (recommended)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='unigram',
+    vocab_size=8000
+)
+```
+
+**Training time**: ~30-40 min for 1GB corpus
+
+## Character coverage
+
+### English/European (0.9995)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='en_corpus.txt',
+    character_coverage=0.9995  # Cover 99.95% of chars
+)
+```
+
+Covers: a-z, A-Z, punctuation, common accents
+
+### CJK (1.0)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='zh_corpus.txt',
+    character_coverage=1.0  # Cover ALL characters
+)
+```
+
+Required for: Chinese, Japanese, Korean
+
+### Multilingual (0.9995-1.0)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='multilingual_corpus.txt',
+    character_coverage=0.9995  # Balance coverage/size
+)
+```
+
+## Vocabulary size selection
+
+| Task | Vocab Size | Rationale |
+|------|------------|-----------|
+| English monolingual | 16k-32k | Standard |
+| Multilingual | 32k-250k | More languages |
+| CJK | 32k-100k | More characters |
+| Code | 16k-32k | Similar to English |
+
+## Normalization rules
+
+### nmt_nfkc (recommended)
+
+```python
+normalization_rule_name='nmt_nfkc'
+```
+
+- NFKC Unicode normalization
+- Whitespace handling
+- **Recommended for most tasks**
+
+### identity (no normalization)
+
+```python
+normalization_rule_name='identity'
+```
+
+- Preserves input exactly
+- Use for code, case-sensitive tasks
+
+### nfkc (standard Unicode)
+
+```python
+normalization_rule_name='nfkc'
+```
+
+- Standard Unicode normalization
+- Less aggressive than nmt_nfkc
+
+## Performance optimization
+
+### Multi-threading
+
+```python
+spm.SentencePieceTrainer.train(
+    input='large_corpus.txt',
+    num_threads=32  # Use all cores
+)
+```
+
+**Speedup**: ~4-8× with 16+ cores
+
+### Sampling input
+
+```python
+spm.SentencePieceTrainer.train(
+    input='huge_corpus.txt',
+    input_sentence_size=10000000,  # Sample 10M sentences
+    shuffle_input_sentence=True
+)
+```
+
+**For very large corpora** (>10GB)
+
+### Extremely large corpus
+
+```python
+spm.SentencePieceTrainer.train(
+    input='massive_corpus.txt',
+    train_extremely_large_corpus=True,  # Enable for >10GB
+    input_sentence_size=100000000
+)
+```
+
+## Best practices
+
+1. **Use Unigram for most tasks** - Better for multilingual
+2. **Set character_coverage=1.0 for CJK** - Required for full coverage
+3. **Use nmt_nfkc normalization** - Works well for most cases
+4. **Add user_defined_symbols for special tokens** - BERT-style tokens
+5. **Enable byte_fallback for robustness** - Handles emojis/rare chars
+6. **Start with vocab_size=32000** - Good default for most tasks
+7. **Use multi-threading** - Speeds up training significantly