@synsci/cli-darwin-x64 1.1.49

package/bin/skills/sentence-transformers/SKILL.md

@@ -0,0 +1,255 @@
+---
+name: sentence-transformers
+description: Framework for state-of-the-art sentence, text, and image embeddings. Provides 5000+ pre-trained models for semantic similarity, clustering, and retrieval. Supports multilingual, domain-specific, and multimodal models. Use for generating embeddings for RAG, semantic search, or similarity tasks. Best for production embedding generation.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Sentence Transformers, Embeddings, Semantic Similarity, RAG, Multilingual, Multimodal, Pre-Trained Models, Clustering, Semantic Search, Production]
+dependencies: [sentence-transformers, transformers, torch]
+---
+
+# Sentence Transformers - State-of-the-Art Embeddings
+
+Python framework for sentence and text embeddings using transformers.
+
+## When to use Sentence Transformers
+
+**Use when:**
+- Need high-quality embeddings for RAG
+- Semantic similarity and search
+- Text clustering and classification
+- Multilingual embeddings (100+ languages)
+- Running embeddings locally (no API)
+- Cost-effective alternative to OpenAI embeddings
+
+**Metrics**:
+- **15,700+ GitHub stars**
+- **5000+ pre-trained models**
+- **100+ languages** supported
+- Based on PyTorch/Transformers
+
+**Use alternatives instead**:
+- **OpenAI Embeddings**: Need API-based, highest quality
+- **Instructor**: Task-specific instructions
+- **Cohere Embed**: Managed service
+
+## Quick start
+
+### Installation
+
+```bash
+pip install sentence-transformers
+```
+
+### Basic usage
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# Load model
+model = SentenceTransformer('all-MiniLM-L6-v2')
+
+# Generate embeddings
+sentences = [
+    "This is an example sentence",
+    "Each sentence is converted to a vector"
+]
+
+embeddings = model.encode(sentences)
+print(embeddings.shape)  # (2, 384)
+
+# Cosine similarity
+from sentence_transformers.util import cos_sim
+similarity = cos_sim(embeddings[0], embeddings[1])
+print(f"Similarity: {similarity.item():.4f}")
+```
+
+## Popular models
+
+### General purpose
+
+```python
+# Fast, good quality (384 dim)
+model = SentenceTransformer('all-MiniLM-L6-v2')
+
+# Better quality (768 dim)
+model = SentenceTransformer('all-mpnet-base-v2')
+
+# Best quality (1024 dim, slower)
+model = SentenceTransformer('all-roberta-large-v1')
+```
+
+### Multilingual
+
+```python
+# 50+ languages
+model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')
+
+# 100+ languages
+model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2')
+```
+
+### Domain-specific
+
+```python
+# Legal domain
+model = SentenceTransformer('nlpaueb/legal-bert-base-uncased')
+
+# Scientific papers
+model = SentenceTransformer('allenai/specter')
+
+# Code
+model = SentenceTransformer('microsoft/codebert-base')
+```
+
+## Semantic search
+
+```python
+from sentence_transformers import SentenceTransformer, util
+
+model = SentenceTransformer('all-MiniLM-L6-v2')
+
+# Corpus
+corpus = [
+    "Python is a programming language",
+    "Machine learning uses algorithms",
+    "Neural networks are powerful"
+]
+
+# Encode corpus
+corpus_embeddings = model.encode(corpus, convert_to_tensor=True)
+
+# Query
+query = "What is Python?"
+query_embedding = model.encode(query, convert_to_tensor=True)
+
+# Find most similar
+hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=3)
+print(hits)
+```
+
+## Similarity computation
+
+```python
+# Cosine similarity
+similarity = util.cos_sim(embedding1, embedding2)
+
+# Dot product
+similarity = util.dot_score(embedding1, embedding2)
+
+# Pairwise cosine similarity
+similarities = util.cos_sim(embeddings, embeddings)
+```
+
+## Batch encoding
+
+```python
+# Efficient batch processing
+sentences = ["sentence 1", "sentence 2", ...] * 1000
+
+embeddings = model.encode(
+    sentences,
+    batch_size=32,
+    show_progress_bar=True,
+    convert_to_tensor=False  # or True for PyTorch tensors
+)
+```
+
+## Fine-tuning
+
+```python
+from sentence_transformers import InputExample, losses
+from torch.utils.data import DataLoader
+
+# Training data
+train_examples = [
+    InputExample(texts=['sentence 1', 'sentence 2'], label=0.8),
+    InputExample(texts=['sentence 3', 'sentence 4'], label=0.3),
+]
+
+train_dataloader = DataLoader(train_examples, batch_size=16)
+
+# Loss function
+train_loss = losses.CosineSimilarityLoss(model)
+
+# Train
+model.fit(
+    train_objectives=[(train_dataloader, train_loss)],
+    epochs=10,
+    warmup_steps=100
+)
+
+# Save
+model.save('my-finetuned-model')
+```
+
+## LangChain integration
+
+```python
+from langchain_community.embeddings import HuggingFaceEmbeddings
+
+embeddings = HuggingFaceEmbeddings(
+    model_name="sentence-transformers/all-mpnet-base-v2"
+)
+
+# Use with vector stores
+from langchain_chroma import Chroma
+
+vectorstore = Chroma.from_documents(
+    documents=docs,
+    embedding=embeddings
+)
+```
+
+## LlamaIndex integration
+
+```python
+from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+
+embed_model = HuggingFaceEmbedding(
+    model_name="sentence-transformers/all-mpnet-base-v2"
+)
+
+from llama_index.core import Settings
+Settings.embed_model = embed_model
+
+# Use in index
+index = VectorStoreIndex.from_documents(documents)
+```
+
+## Model selection guide
+
+| Model | Dimensions | Speed | Quality | Use Case |
+|-------|------------|-------|---------|----------|
+| all-MiniLM-L6-v2 | 384 | Fast | Good | General, prototyping |
+| all-mpnet-base-v2 | 768 | Medium | Better | Production RAG |
+| all-roberta-large-v1 | 1024 | Slow | Best | High accuracy needed |
+| paraphrase-multilingual | 768 | Medium | Good | Multilingual |
+
+## Best practices
+
+1. **Start with all-MiniLM-L6-v2** - Good baseline
+2. **Normalize embeddings** - Better for cosine similarity
+3. **Use GPU if available** - 10× faster encoding
+4. **Batch encoding** - More efficient
+5. **Cache embeddings** - Expensive to recompute
+6. **Fine-tune for domain** - Improves quality
+7. **Test different models** - Quality varies by task
+8. **Monitor memory** - Large models need more RAM
+
+## Performance
+
+| Model | Speed (sentences/sec) | Memory | Dimension |
+|-------|----------------------|---------|-----------|
+| MiniLM | ~2000 | 120MB | 384 |
+| MPNet | ~600 | 420MB | 768 |
+| RoBERTa | ~300 | 1.3GB | 1024 |
+
+## Resources
+
+- **GitHub**: https://github.com/UKPLab/sentence-transformers ⭐ 15,700+
+- **Models**: https://huggingface.co/sentence-transformers
+- **Docs**: https://www.sbert.net
+- **License**: Apache 2.0
+
+

package/bin/skills/sentence-transformers/references/models.md

@@ -0,0 +1,123 @@
+# Sentence Transformers Models Guide
+
+Guide to selecting and using sentence-transformers models.
+
+## Top recommended models
+
+### General purpose
+
+**all-MiniLM-L6-v2** (Default recommendation)
+- Dimensions: 384
+- Speed: ~2000 sentences/sec
+- Quality: Good
+- Use: Prototyping, general tasks
+
+**all-mpnet-base-v2** (Best quality)
+- Dimensions: 768
+- Speed: ~600 sentences/sec
+- Quality: Better
+- Use: Production RAG
+
+**all-roberta-large-v1** (Highest quality)
+- Dimensions: 1024
+- Speed: ~300 sentences/sec
+- Quality: Best
+- Use: When accuracy critical
+
+### Multilingual (50+ languages)
+
+**paraphrase-multilingual-MiniLM-L12-v2**
+- Languages: 50+
+- Dimensions: 384
+- Speed: Fast
+- Use: Multilingual semantic search
+
+**paraphrase-multilingual-mpnet-base-v2**
+- Languages: 50+
+- Dimensions: 768
+- Speed: Medium
+- Use: Better multilingual quality
+
+**LaBSE** (109 languages)
+- Languages: 109
+- Dimensions: 768
+- Speed: Medium
+- Use: Maximum language coverage
+
+### Domain-specific
+
+**allenai/specter** (Scientific papers)
+- Domain: Academic papers
+- Use: Paper similarity, citations
+
+**nlpaueb/legal-bert-base-uncased** (Legal)
+- Domain: Legal documents
+- Use: Legal document analysis
+
+**microsoft/codebert-base** (Code)
+- Domain: Source code
+- Use: Code similarity, search
+
+## Model selection matrix
+
+| Task | Model | Dimensions | Speed | Quality |
+|------|-------|------------|-------|---------|
+| Quick prototyping | MiniLM-L6 | 384 | Fast | Good |
+| Production RAG | mpnet-base | 768 | Medium | Better |
+| Highest accuracy | roberta-large | 1024 | Slow | Best |
+| Multilingual | paraphrase-multi-mpnet | 768 | Medium | Good |
+| Scientific papers | specter | 768 | Medium | Domain |
+| Legal docs | legal-bert | 768 | Medium | Domain |
+
+## Performance benchmarks
+
+### Speed comparison (CPU)
+
+| Model | Sentences/sec | Memory |
+|-------|---------------|--------|
+| MiniLM-L6 | 2000 | 120 MB |
+| MPNet-base | 600 | 420 MB |
+| RoBERTa-large | 300 | 1.3 GB |
+
+### Quality comparison (STS Benchmark)
+
+| Model | Cosine Similarity | Spearman |
+|-------|-------------------|----------|
+| MiniLM-L6 | 82.4 | - |
+| MPNet-base | 84.1 | - |
+| RoBERTa-large | 85.4 | - |
+
+## Usage examples
+
+### Load and use model
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# Load model
+model = SentenceTransformer('all-mpnet-base-v2')
+
+# Generate embeddings
+sentences = ["This is a sentence", "This is another sentence"]
+embeddings = model.encode(sentences)
+```
+
+### Compare different models
+
+```python
+models = {
+    'MiniLM': 'all-MiniLM-L6-v2',
+    'MPNet': 'all-mpnet-base-v2',
+    'RoBERTa': 'all-roberta-large-v1'
+}
+
+for name, model_name in models.items():
+    model = SentenceTransformer(model_name)
+    embeddings = model.encode(["Test sentence"])
+    print(f"{name}: {embeddings.shape}")
+```
+
+## Resources
+
+- **Models**: https://huggingface.co/sentence-transformers
+- **Docs**: https://www.sbert.net/docs/pretrained_models.html

package/bin/skills/sentencepiece/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: sentencepiece
+description: Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Tokenization, SentencePiece, Language-Independent, BPE, Unigram, Multilingual, CJK Languages, Unicode, Deterministic, Google]
+dependencies: [sentencepiece, transformers]
+---
+
+# SentencePiece - Language-Independent Tokenization
+
+Unsupervised tokenizer that works on raw text without language-specific preprocessing.
+
+## When to use SentencePiece
+
+**Use SentencePiece when:**
+- Building multilingual models (no language-specific rules)
+- Working with CJK languages (Chinese, Japanese, Korean)
+- Need reproducible tokenization (deterministic vocabulary)
+- Want to train on raw text (no pre-tokenization needed)
+- Require lightweight deployment (6MB memory, 50k sentences/sec)
+
+**Performance**:
+- **Speed**: 50,000 sentences/sec
+- **Memory**: ~6MB for loaded model
+- **Languages**: All (language-independent)
+
+**Use alternatives instead**:
+- **HuggingFace Tokenizers**: Faster training, more flexibility
+- **tiktoken**: OpenAI models (GPT-3.5/4)
+- **BERT WordPiece**: English-centric tasks
+
+## Quick start
+
+### Installation
+
+```bash
+# Python
+pip install sentencepiece
+
+# C++ (requires CMake)
+git clone https://github.com/google/sentencepiece.git
+cd sentencepiece
+mkdir build && cd build
+cmake .. && make -j $(nproc)
+sudo make install
+```
+
+### Train model
+
+```bash
+# Command-line (BPE with 8000 vocab)
+spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe
+
+# Python API
+import sentencepiece as spm
+
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='m',
+    vocab_size=8000,
+    model_type='bpe'
+)
+```
+
+**Training time**: ~1-2 minutes for 100MB corpus
+
+### Encode and decode
+
+```python
+import sentencepiece as spm
+
+# Load model
+sp = spm.SentencePieceProcessor(model_file='m.model')
+
+# Encode to pieces
+pieces = sp.encode('This is a test', out_type=str)
+print(pieces)  # ['▁This', '▁is', '▁a', '▁test']
+
+# Encode to IDs
+ids = sp.encode('This is a test', out_type=int)
+print(ids)  # [284, 47, 11, 1243]
+
+# Decode
+text = sp.decode(ids)
+print(text)  # "This is a test"
+```
+
+## Language-independent design
+
+### Whitespace as symbol (▁)
+
+```python
+text = "Hello world"
+pieces = sp.encode(text, out_type=str)
+print(pieces)  # ['▁Hello', '▁world']
+
+# Decode preserves spaces
+decoded = sp.decode_pieces(pieces)
+print(decoded)  # "Hello world"
+```
+
+**Key principle**: Treat text as raw Unicode, whitespace = ▁ (meta symbol)
+
+## Tokenization algorithms
+
+### BPE (Byte-Pair Encoding)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='bpe_model',
+    vocab_size=16000,
+    model_type='bpe'
+)
+```
+
+**Used by**: mBART
+
+### Unigram (default)
+
+```python
+spm.SentencePieceTrainer.train(
+    input='data.txt',
+    model_prefix='unigram_model',
+    vocab_size=8000,
+    model_type='unigram'
+)
+```
+
+**Used by**: T5, ALBERT, XLNet
+
+## Training configuration
+
+### Essential parameters
+
+```python
+spm.SentencePieceTrainer.train(
+    input='corpus.txt',
+    model_prefix='m',
+    vocab_size=32000,
+    model_type='unigram',
+    character_coverage=0.9995,  # 1.0 for CJK
+    user_defined_symbols=['[SEP]', '[CLS]'],
+    unk_piece='<unk>',
+    num_threads=16
+)
+```
+
+### Character coverage
+
+| Language Type | Coverage | Rationale |
+|---------------|----------|-----------|
+| English       | 0.9995   | Most common chars |
+| CJK (Chinese) | 1.0      | All characters needed |
+| Multilingual  | 0.9995   | Balance |
+
+## Encoding options
+
+### Subword regularization
+
+```python
+# Sample different tokenizations
+for _ in range(3):
+    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)
+    print(pieces)
+
+# Output (different each time):
+# ['▁token', 'ization']
+# ['▁tok', 'en', 'ization']
+```
+
+**Use case**: Data augmentation for robustness.
+
+## Common patterns
+
+### T5-style training
+
+```python
+spm.SentencePieceTrainer.train(
+    input='c4_corpus.txt',
+    model_prefix='t5',
+    vocab_size=32000,
+    model_type='unigram',
+    user_defined_symbols=[f'<extra_id_{i}>' for i in range(100)],
+    unk_id=2,
+    eos_id=1,
+    pad_id=0
+)
+```
+
+### Integration with transformers
+
+```python
+from transformers import T5Tokenizer
+
+# T5 uses SentencePiece internally
+tokenizer = T5Tokenizer.from_pretrained('t5-base')
+inputs = tokenizer('translate English to French: Hello', return_tensors='pt')
+```
+
+## Performance benchmarks
+
+### Training speed
+
+| Corpus | BPE (16k) | Unigram (8k) |
+|--------|-----------|--------------|
+| 100 MB | 1-2 min   | 3-4 min      |
+| 1 GB   | 10-15 min | 30-40 min    |
+
+### Tokenization speed
+
+- **SentencePiece**: 50,000 sentences/sec
+- **HF Tokenizers**: 200,000 sentences/sec (4× faster)
+
+## Supported models
+
+**T5 family**: `t5-base`, `t5-large` (32k vocab, Unigram)
+**ALBERT**: `albert-base-v2` (30k vocab, Unigram)
+**XLNet**: `xlnet-base-cased` (32k vocab, Unigram)
+**mBART**: `facebook/mbart-large-50` (250k vocab, BPE)
+
+## References
+
+- **[Training Guide](references/training.md)** - Detailed options, corpus preparation
+- **[Algorithms](references/algorithms.md)** - BPE vs Unigram, subword regularization
+
+## Resources
+
+- **GitHub**: https://github.com/google/sentencepiece ⭐ 10,000+
+- **Paper**: https://arxiv.org/abs/1808.06226 (EMNLP 2018)
+- **Version**: 0.2.0+
+
+