@r1z4x/tezgah 1.0.0

package/skills/ai-research/02-tokenization/sentencepiece/references/training.md

@@ -0,0 +1,305 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 02-tokenization/sentencepiece/references/training.md; MIT (c) Orchestra Research -->
+# SentencePiece Training Guide
+
+Complete guide to training SentencePiece models.
+
+## Training workflow
+
+### Step 1: Prepare corpus
+
+```bash
+# Plain text file, one sentence per line (recommended)
+cat corpus.txt
+# Hello world
+# This is a test
+# SentencePiece is language-independent
+
+# Or use raw text (SentencePiece handles sentence splitting)
+```
+
+### Step 2: Train model
+
+**Command-line**:
+```bash
+spm_train \
+  --input=corpus.txt \
+  --model_prefix=m \
+  --vocab_size=8000 \
+  --model_type=unigram \
+  --character_coverage=0.9995
+```
+
+**Python API**:
+```python
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=8000,
+    model_type='unigram'
+)
+```
+
+**Output**: `m.model` (binary), `m.vocab` (text vocabulary)
+
+### Step 3: Load and use
+
+```python
+sp = spm.SentencePieceProcessor(model_file='m.model')
+pieces = sp.encode('Test sentence', out_type=str)
+```
+
+## Training parameters
+
+### Core parameters
+
+```python
+spm.SentencePieceTrainer.train(
+    # Required
+    input='corpus.txt',           # Input corpus
+    model_prefix='output',        # Output prefix
+    vocab_size=8000,              # Target vocabulary size
+
+    # Algorithm
+    model_type='unigram',         # 'unigram', 'bpe', 'char', 'word'
+
+    # Coverage
+    character_coverage=0.9995,    # 0.9995 for most, 1.0 for CJK
+
+    # Normalization
+    normalization_rule_name='nmt_nfkc',  # 'nmt_nfkc', 'nfkc', 'identity'
+
+    # Performance
+    num_threads=16,               # Training threads
+    input_sentence_size=10000000  # Max sentences to load
+)
+```
+
+### Special tokens
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+
+    # Control symbols (special tokens for model control)
+    control_symbols=['<s>', '</s>', '<pad>'],
+
+    # User-defined symbols (never split)
+    user_defined_symbols=['[MASK]', '[SEP]', '[CLS]'],
+
+    # Special token pieces
+    unk_piece='<unk>',
+    bos_piece='<s>',
+    eos_piece='</s>',
+    pad_piece='<pad>',
+
+    # Special token IDs
+    unk_id=0,
+    bos_id=1,
+    eos_id=2,
+    pad_id=3
+)
+```
+
+### Advanced options
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+
+    # Byte fallback (handle unknown chars)
+    byte_fallback=True,
+
+    # Digit handling
+    split_digits=True,            # Split digits individually
+
+    # Script splitting
+    split_by_unicode_script=True, # Split by Unicode script
+    split_by_whitespace=True,     # Split by whitespace
+
+    # Length constraints
+    max_sentencepiece_length=16,  # Max token length
+
+    # Rare word handling
+    min_frequency=2,              # Min frequency for token
+
+    # Training size
+    input_sentence_size=10000000, # Max sentences
+    shuffle_input_sentence=True,  # Shuffle training data
+
+    # Seed
+    seed_sentencepiece_size=1000000  # Seed vocab size
+)
+```
+
+## Training from Python iterator
+
+```python
+import sentencepiece as spm
+from datasets import load_dataset
+
+# Load dataset
+dataset = load_dataset('wikitext', 'wikitext-103-raw-v1', split='train')
+
+# Create iterator
+def corpus_iterator():
+    for example in dataset:
+        if example['text'].strip():
+            yield example['text']
+
+# Train from iterator
+spm.SentencePieceTrainer.train(
+    sentence_iterator=corpus_iterator(),
+    model_prefix='wiki',
+    vocab_size=32000,
+    model_type='unigram'
+)
+```
+
+## Model types
+
+### BPE
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='bpe',
+    vocab_size=16000
+)
+```
+
+**Training time**: ~10-15 min for 1GB corpus
+
+### Unigram (recommended)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_type='unigram',
+    vocab_size=8000
+)
+```
+
+**Training time**: ~30-40 min for 1GB corpus
+
+## Character coverage
+
+### English/European (0.9995)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='en_corpus.txt',
+    character_coverage=0.9995  # Cover 99.95% of chars
+)
+```
+
+Covers: a-z, A-Z, punctuation, common accents
+
+### CJK (1.0)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='zh_corpus.txt',
+    character_coverage=1.0  # Cover ALL characters
+)
+```
+
+Required for: Chinese, Japanese, Korean
+
+### Multilingual (0.9995-1.0)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='multilingual_corpus.txt',
+    character_coverage=0.9995  # Balance coverage/size
+)
+```
+
+## Vocabulary size selection
+
+| Task | Vocab Size | Rationale |
+|------|------------|-----------|
+| English monolingual | 16k-32k | Standard |
+| Multilingual | 32k-250k | More languages |
+| CJK | 32k-100k | More characters |
+| Code | 16k-32k | Similar to English |
+
+## Normalization rules
+
+### nmt_nfkc (recommended)
+
+```python
+normalization_rule_name='nmt_nfkc'
+```
+
+- NFKC Unicode normalization
+- Whitespace handling
+- **Recommended for most tasks**
+
+### identity (no normalization)
+
+```python
+normalization_rule_name='identity'
+```
+
+- Preserves input exactly
+- Use for code, case-sensitive tasks
+
+### nfkc (standard Unicode)
+
+```python
+normalization_rule_name='nfkc'
+```
+
+- Standard Unicode normalization
+- Less aggressive than nmt_nfkc
+
+## Performance optimization
+
+### Multi-threading
+
+```python
+spm.SentencePieceTrainer.train(
+    input='large_corpus.txt',
+    num_threads=32  # Use all cores
+)
+```
+
+**Speedup**: ~4-8× with 16+ cores
+
+### Sampling input
+
+```python
+spm.SentencePieceTrainer.train(
+    input='huge_corpus.txt',
+    input_sentence_size=10000000,  # Sample 10M sentences
+    shuffle_input_sentence=True
+)
+```
+
+**For very large corpora** (>10GB)
+
+### Extremely large corpus
+
+```python
+spm.SentencePieceTrainer.train(
+    input='massive_corpus.txt',
+    train_extremely_large_corpus=True,  # Enable for >10GB
+    input_sentence_size=100000000
+)
+```
+
+## Best practices
+
+1. **Use Unigram for most tasks** - Better for multilingual
+2. **Set character_coverage=1.0 for CJK** - Required for full coverage
+3. **Use nmt_nfkc normalization** - Works well for most cases
+4. **Add user_defined_symbols for special tokens** - BERT-style tokens
+5. **Enable byte_fallback for robustness** - Handles emojis/rare chars
+6. **Start with vocab_size=32000** - Good default for most tasks
+7. **Use multi-threading** - Speeds up training significantly

package/skills/ai-research/03-fine-tuning/axolotl/SKILL.md

@@ -0,0 +1,158 @@
+---
+name: axolotl
+description: Expert guidance for fine-tuning LLMs with Axolotl - YAML configs, 100+ models, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+tags: [Fine-Tuning, Axolotl, LLM, LoRA, QLoRA, DPO, KTO, ORPO, GRPO, YAML, HuggingFace, DeepSpeed, Multimodal]
+dependencies: [axolotl, torch, transformers, datasets, peft, accelerate, deepspeed]
+---
+
+# Axolotl Skill
+
+Comprehensive assistance with axolotl development, generated from official documentation.
+
+## When to Use This Skill
+
+This skill should be triggered when:
+- Working with axolotl
+- Asking about axolotl features or APIs
+- Implementing axolotl solutions
+- Debugging axolotl code
+- Learning axolotl best practices
+
+## Quick Reference
+
+### Common Patterns
+
+**Pattern 1:** To validate that acceptable data transfer speeds exist for your training job, running NCCL Tests can help pinpoint bottlenecks, for example:
+
+```
+./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
+```
+
+**Pattern 2:** Configure your model to use FSDP in the Axolotl yaml. For example:
+
+```
+fsdp_version: 2
+fsdp_config:
+  offload_params: true
+  state_dict_type: FULL_STATE_DICT
+  auto_wrap_policy: TRANSFORMER_BASED_WRAP
+  transformer_layer_cls_to_wrap: LlamaDecoderLayer
+  reshard_after_forward: true
+```
+
+**Pattern 3:** The context_parallel_size should be a divisor of the total number of GPUs. For example:
+
+```
+context_parallel_size
+```
+
+**Pattern 4:** For example: - With 8 GPUs and no sequence parallelism: 8 different batches processed per step - With 8 GPUs and context_parallel_size=4: Only 2 different batches processed per step (each split across 4 GPUs) - If your per-GPU micro_batch_size is 2, the global batch size decreases from 16 to 4
+
+```
+context_parallel_size=4
+```
+
+**Pattern 5:** Setting save_compressed: true in your configuration enables saving models in a compressed format, which: - Reduces disk space usage by approximately 40% - Maintains compatibility with vLLM for accelerated inference - Maintains compatibility with llmcompressor for further optimization (example: quantization)
+
+```
+save_compressed: true
+```
+
+**Pattern 6:** Note It is not necessary to place your integration in the integrations folder. It can be in any location, so long as it’s installed in a package in your python env. See this repo for an example: https://github.com/axolotl-ai-cloud/diff-transformer
+
+```
+integrations
+```
+
+**Pattern 7:** Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
+
+```
+utils.trainer.drop_long_seq(sample, sequence_len=2048, min_sequence_len=2)
+```
+
+### Example Code Patterns
+
+**Example 1** (python):
+```python
+cli.cloud.modal_.ModalCloud(config, app=None)
+```
+
+**Example 2** (python):
+```python
+cli.cloud.modal_.run_cmd(cmd, run_folder, volumes=None)
+```
+
+**Example 3** (python):
+```python
+core.trainers.base.AxolotlTrainer(
+    *_args,
+    bench_data_collator=None,
+    eval_data_collator=None,
+    dataset_tags=None,
+    **kwargs,
+)
+```
+
+**Example 4** (python):
+```python
+core.trainers.base.AxolotlTrainer.log(logs, start_time=None)
+```
+
+**Example 5** (python):
+```python
+prompt_strategies.input_output.RawInputOutputPrompter()
+```
+
+## Reference Files
+
+This skill includes comprehensive documentation in `references/`:
+
+- **api.md** - Api documentation
+- **dataset-formats.md** - Dataset-Formats documentation
+- **other.md** - Other documentation
+
+Use `view` to read specific reference files when detailed information is needed.
+
+## Working with This Skill
+
+### For Beginners
+Start with the getting_started or tutorials reference files for foundational concepts.
+
+### For Specific Features
+Use the appropriate category reference file (api, guides, etc.) for detailed information.
+
+### For Code Examples
+The quick reference section above contains common patterns extracted from the official docs.
+
+## Resources
+
+### references/
+Organized documentation extracted from official sources. These files contain:
+- Detailed explanations
+- Code examples with language annotations
+- Links to original documentation
+- Table of contents for quick navigation
+
+### scripts/
+Add helper scripts here for common automation tasks.
+
+### assets/
+Add templates, boilerplate, or example projects here.
+
+## Notes
+
+- This skill was automatically generated from official documentation
+- Reference files preserve the structure and examples from source docs
+- Code examples include language detection for better syntax highlighting
+- Quick reference patterns are extracted from common usage examples in the docs
+
+## Updating
+
+To refresh this skill with updated documentation:
+1. Re-run the scraper with the same configuration
+2. The skill will be rebuilt with the latest information
+
+