@synsci/cli-darwin-x64-baseline 1.1.73 → 1.1.74

package/bin/skills/model-economics/SKILL.md

@@ -0,0 +1,238 @@
+---
+name: model-economics
+description: Cost modeling and ROI analysis for specialized LLM development. Use when deciding whether to train a custom model, estimating total cost, or calculating break-even vs frontier APIs. Covers training costs, inference costs, and time-to-ROI projections.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Economics, Cost Analysis, ROI, Break-Even, Training Cost, Inference Cost, Model Flywheel, Business Case]
+dependencies: []
+---
+
+# Model Economics: Cost Analysis for LLM Specialization
+
+## When to Use This Skill
+
+Use this skill as the **first step** of any flywheel assessment to determine:
+- Is it worth training a specialized model?
+- What will the total investment be?
+- When will the specialized model break even vs frontier APIs?
+- What's the ongoing cost comparison?
+
+## Frontier API Cost Model
+
+**Important**: Use `websearch` to get current pricing — these change frequently. Reference tiers as of early 2026:
+
+### Premium Tier ($5-25/M output tokens)
+| Model | Input ($/M) | Output ($/M) | Notes |
+|-------|-------------|--------------|-------|
+| Claude Opus 4.6 | $5.00 | $25.00 | Highest capability |
+| GPT-5.2 Pro | $21.00 | $168.00 | Reasoning-heavy |
+
+**Savings potential**: Highest. A specialized 8B model costs ~$0.05-0.15/M tokens to serve = 100-500x cheaper.
+
+### Standard Tier ($1.75-14/M output tokens)
+| Model | Input ($/M) | Output ($/M) | Notes |
+|-------|-------------|--------------|-------|
+| GPT-5.2 Chat | $1.75 | $14.00 | Most popular |
+| Claude Sonnet 4.5 | $3.00 | $15.00 | Good balance |
+| Qwen3 Max | $1.20 | $6.00 | Cost-effective |
+
+**Savings potential**: Strong at volume. Break-even is achievable with moderate traffic.
+
+### Budget Tier ($0.10-3/M output tokens)
+| Model | Input ($/M) | Output ($/M) | Notes |
+|-------|-------------|--------------|-------|
+| Gemini 3 Flash | $0.50 | $3.00 | Fast, cheap |
+| GLM 4.7 | $0.40 | $1.50 | Very cost-effective |
+| Mistral Small | $0.10 | $0.30 | Cheapest commercial |
+
+**Savings potential**: Harder to beat on cost alone. Specialize for quality, not cost.
+
+### Free / Near-Free
+Many open-weight models available on OpenRouter at $0. If you only need basic capability, self-hosting these is essentially free beyond GPU cost.
+
+## Training Cost Estimation
+
+### Tinker (Managed LoRA)
+Cheapest option for supported models. No GPU management.
+
+| Model Size | Method | Cost/1K examples | Duration |
+|------------|--------|-------------------|----------|
+| 8B | LoRA | ~$2-5 | 15-30 min |
+| 14B | LoRA | ~$5-10 | 30-60 min |
+| 32B | LoRA | ~$10-25 | 1-3 hours |
+| 70B | LoRA | ~$25-60 | 3-8 hours |
+
+Load `tinker-training-cost` skill for exact per-token rates.
+
+### Modal (Serverless GPU)
+
+| GPU | Cost/Hour | Best For |
+|-----|-----------|----------|
+| A100-40GB | ~$1.10 | 7-14B LoRA, 7B full |
+| A100-80GB | ~$1.60 | 14-32B LoRA, 14B full |
+| H100-80GB | ~$3.25 | 32-70B LoRA, large batches |
+
+**Typical training runs**:
+- 8B LoRA, 5K examples, 3 epochs: ~1-2 hours on A100 = $1-3
+- 14B LoRA, 10K examples, 3 epochs: ~3-5 hours on A100-80GB = $5-8
+- 70B LoRA, 10K examples, 2 epochs: ~8-12 hours on H100 = $26-39
+
+### TensorPool (Dedicated GPU)
+
+| GPU | Cost/Hour | Best For |
+|-----|-----------|----------|
+| A100-80GB | ~$1.50 | Standard training |
+| H200-141GB | ~$3.00 | Large models, fast training |
+| B200-192GB | ~$4.50 | Cutting-edge, maximum throughput |
+
+Best for: Multi-day training, large-scale RL, dedicated inference.
+
+## Inference Cost Comparison
+
+The key calculation: what does it cost to serve the specialized model vs the frontier API?
+
+### Self-Hosted Inference Cost
+
+```python
+def inference_cost_per_million_tokens(gpu_cost_per_hour, tokens_per_second):
+    """Calculate cost per million tokens for self-hosted inference.
+
+    Args:
+        gpu_cost_per_hour: GPU rental cost (e.g., $3.25 for H100)
+        tokens_per_second: Model throughput (varies by model size and batch)
+    """
+    tokens_per_hour = tokens_per_second * 3600
+    cost_per_token = gpu_cost_per_hour / tokens_per_hour
+    cost_per_million = cost_per_token * 1_000_000
+    return cost_per_million
+```
+
+### Reference Throughput (vLLM, single GPU)
+
+| Model Size | GPU | Tokens/sec (batch) | Cost/M tokens |
+|------------|-----|-------------------|---------------|
+| 8B | A100-80GB | ~3,000-5,000 | $0.09-0.15 |
+| 8B | H100-80GB | ~5,000-8,000 | $0.11-0.18 |
+| 14B | A100-80GB | ~1,500-3,000 | $0.15-0.30 |
+| 14B | H100-80GB | ~3,000-5,000 | $0.18-0.30 |
+| 32B | H100-80GB | ~800-1,500 | $0.60-1.10 |
+| 70B | 2xH100 | ~500-1,000 | $1.80-3.60 |
+
+**Key insight**: A specialized 8B model on Modal/TensorPool costs **$0.05-0.15/M tokens** vs **$5-25/M** for premium frontier = **50-500x cheaper per request**.
+
+## Break-Even Calculator
+
+```python
+def break_even_analysis(
+    monthly_api_spend: float,
+    training_cost: float,
+    monthly_inference_cost: float,
+    setup_hours: float = 40,
+    hourly_rate: float = 100,
+):
+    """Calculate break-even timeline for model specialization.
+
+    Args:
+        monthly_api_spend: Current monthly frontier API cost
+        training_cost: One-time training cost (GPU + data prep)
+        monthly_inference_cost: Monthly cost to serve specialized model
+        setup_hours: Engineering time to set up pipeline
+        hourly_rate: Engineering cost per hour
+    """
+    engineering_cost = setup_hours * hourly_rate
+    total_upfront = training_cost + engineering_cost
+    monthly_savings = monthly_api_spend - monthly_inference_cost
+
+    if monthly_savings <= 0:
+        return {
+            "viable": False,
+            "reason": "Specialized model costs more to serve than frontier API",
+            "monthly_savings": monthly_savings,
+        }
+
+    months_to_break_even = total_upfront / monthly_savings
+
+    return {
+        "viable": True,
+        "total_upfront_cost": total_upfront,
+        "training_cost": training_cost,
+        "engineering_cost": engineering_cost,
+        "monthly_api_spend": monthly_api_spend,
+        "monthly_inference_cost": monthly_inference_cost,
+        "monthly_savings": monthly_savings,
+        "months_to_break_even": round(months_to_break_even, 1),
+        "annual_savings": monthly_savings * 12 - total_upfront,
+        "year_2_savings": monthly_savings * 12,  # Fully amortized
+    }
+
+
+# Example: Replace GPT-4o for a specific task
+result = break_even_analysis(
+    monthly_api_spend=5000,       # $5K/month on GPT-4o
+    training_cost=500,            # $500 for LoRA fine-tune + data prep
+    monthly_inference_cost=200,   # $200/month on Modal (8B model)
+    setup_hours=40,               # 1 week of engineering
+    hourly_rate=100,              # $100/hr engineer
+)
+# Result: Break-even in ~1.1 months, $52K annual savings after year 1
+```
+
+## ROI Timeline Template
+
+| Month | Frontier Cost | Specialized Cost | Cumulative Savings | Notes |
+|-------|--------------|-----------------|-------------------|-------|
+| 0 | — | $4,500 upfront | -$4,500 | Training + engineering |
+| 1 | $5,000 | $200 | $300 | First month live |
+| 2 | $5,000 | $200 | $5,100 | Break-even |
+| 3 | $5,000 | $200 | $9,900 | |
+| 6 | $5,000 | $200 | $24,300 | |
+| 12 | $5,000 | $300 | $52,200 | Includes retrain cost |
+
+## Decision Framework
+
+### Strong Case for Specialization
+- Monthly API spend > $1,000
+- Constrained task (classification, extraction, formatting, domain Q&A)
+- Production data available (logs, corrections, accept/reject signals)
+- Quality requirements are well-defined and measurable
+- Task doesn't change frequently
+
+### Weak Case for Specialization
+- Monthly API spend < $500
+- Diverse, open-ended tasks (general assistant)
+- No production data to train on
+- Rapid model improvement expected (new frontier models releasing soon)
+- Task requires reasoning at the frontier level
+- Small team with no ML engineering capacity
+
+### Hybrid Approach
+Route easy requests to specialized model, hard ones to frontier:
+- **90% of requests** → Specialized 8B (cheap, fast)
+- **10% of requests** → Frontier fallback (complex, novel)
+- **Result**: 80-90% cost reduction while maintaining quality on hard cases
+
+## Quick Assessment Questions
+
+Ask the user these to determine viability:
+
+1. **What's your monthly frontier API spend?**
+   - < $500: Probably not worth it (low ROI)
+   - $500-2K: Worth investigating
+   - $2K+: Strong candidate
+   - $10K+: Almost certainly worth it
+
+2. **How constrained is the task?**
+   - Single task (classification, extraction): Great fit
+   - Few related tasks: Good fit
+   - Many diverse tasks: Harder to specialize
+
+3. **What production data do you have?**
+   - API logs with user feedback: Excellent
+   - API logs without feedback: Good (can distill)
+   - No production data: Need synthetic bootstrap
+
+4. **What's your quality bar?**
+   - Must match frontier exactly: Harder, may need bigger model
+   - 90% of frontier quality is fine: 8B LoRA likely sufficient
+   - Task-specific metrics (accuracy, format): Easiest to optimize

package/bin/skills/training-data-pipeline/SKILL.md

@@ -0,0 +1,427 @@
+---
+name: training-data-pipeline
+description: Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train/eval splitting.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Training Data, Data Pipeline, Fine-Tuning, Distillation, Synthetic Data, Production Data, JSONL, Data Quality]
+dependencies: [datasets, transformers, openai]
+---
+
+# Training Data Pipeline
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- **Format production logs** into SFT training data (API logs, user corrections, accept/reject signals)
+- **Distill from frontier models** using batch APIs (OpenAI, Anthropic) to label production inputs
+- **Bootstrap synthetic data** when fewer than 1000 real examples exist
+- **Validate data quality** before training (dedup, schema check, diversity metrics)
+- **Split data** into train/eval sets with production data reserved for evaluation
+
+### Three Data Paths
+
+| Path | When to Use | Data Source | Cost |
+|------|-------------|-------------|------|
+| A) Production data | Have API logs or user feedback | Your own production systems | Free (already collected) |
+| B) Frontier distillation | Have production inputs but no labels | OpenAI/Anthropic batch APIs | ~50% of real-time API cost |
+| C) Synthetic bootstrap | < 1000 real examples | Frontier model generation | Varies by volume |
+
+**Always prefer Path A** — production data is the moat competitors can't replicate.
+
+## JSONL Chat Format
+
+All training platforms (Tinker, Unsloth, TRL, Axolotl) accept this standard chat format:
+
+```jsonl
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]}
+{"messages": [{"role": "user", "content": "Translate to French: Hello"}, {"role": "assistant", "content": "Bonjour"}]}
+```
+
+### Format Rules
+- One JSON object per line, no trailing commas
+- `messages` array with `role` and `content` fields
+- Roles: `system` (optional, first only), `user`, `assistant` (alternating)
+- Multi-turn: alternate user/assistant pairs within a single messages array
+- UTF-8 encoding, no BOM
+- `assistant` messages are the training targets — everything else is context
+
+### Platform-Specific Notes
+
+**Tinker**: Standard chat format above. Max 32K tokens per example. System message optional.
+
+**Unsloth/TRL**: Same format. Also accepts `{"prompt": "...", "completion": "..."}` for simple pairs. Chat format preferred for multi-turn.
+
+**Axolotl**: Supports multiple formats via config. Recommend `chat_template` type with standard JSONL.
+
+## Path A: Production Data Collection
+
+### From API Logs
+
+If you log API requests/responses, convert them directly:
+
+```python
+import json
+
+def api_log_to_training(log_entry):
+    """Convert an API request/response log to training format."""
+    messages = []
+
+    # Add system prompt if present
+    if log_entry.get("system_prompt"):
+        messages.append({
+            "role": "system",
+            "content": log_entry["system_prompt"]
+        })
+
+    # Add the user's input
+    messages.append({
+        "role": "user",
+        "content": log_entry["user_input"]
+    })
+
+    # Add the response (use corrected version if available)
+    response = log_entry.get("corrected_response") or log_entry["api_response"]
+    messages.append({
+        "role": "assistant",
+        "content": response
+    })
+
+    return {"messages": messages}
+
+# Process logs
+with open("api_logs.jsonl") as f, open("training_data.jsonl", "w") as out:
+    for line in f:
+        log = json.loads(line)
+        example = api_log_to_training(log)
+        out.write(json.dumps(example) + "\n")
+```
+
+### From User Corrections
+
+User corrections (edits to model output) are the highest-quality training signal:
+
+```python
+def correction_to_training(original_input, corrected_output, system_prompt=None):
+    """Convert a user correction into a training example.
+    The corrected output becomes the training target."""
+    messages = []
+    if system_prompt:
+        messages.append({"role": "system", "content": system_prompt})
+    messages.append({"role": "user", "content": original_input})
+    messages.append({"role": "assistant", "content": corrected_output})
+    return {"messages": messages}
+```
+
+### From Accept/Reject Signals
+
+If users accept or reject model outputs, use accepted outputs as positive examples:
+
+```python
+def filter_accepted(logs):
+    """Keep only examples where user accepted the output."""
+    accepted = []
+    for log in logs:
+        if log.get("user_action") == "accepted":
+            accepted.append({
+                "messages": [
+                    {"role": "user", "content": log["input"]},
+                    {"role": "assistant", "content": log["output"]}
+                ]
+            })
+    return accepted
+```
+
+## Path B: Frontier Distillation
+
+Use frontier models to label your production inputs. Best when you have real inputs but no gold labels.
+
+### OpenAI Batch API (50% discount)
+
+```python
+import json
+
+def create_batch_file(inputs, system_prompt, model="gpt-4o"):
+    """Create a batch file for OpenAI Batch API."""
+    requests = []
+    for i, user_input in enumerate(inputs):
+        requests.append({
+            "custom_id": f"request-{i}",
+            "method": "POST",
+            "url": "/v1/chat/completions",
+            "body": {
+                "model": model,
+                "messages": [
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_input}
+                ],
+                "max_tokens": 4096
+            }
+        })
+
+    with open("batch_input.jsonl", "w") as f:
+        for req in requests:
+            f.write(json.dumps(req) + "\n")
+    return "batch_input.jsonl"
+
+# Submit batch
+# openai api batches create -i batch_input.jsonl -e /v1/chat/completions -c 24h
+```
+
+### Anthropic Batch API
+
+```python
+import anthropic
+
+client = anthropic.Anthropic()
+
+def create_anthropic_batch(inputs, system_prompt, model="claude-sonnet-4-5-20250929"):
+    """Create batch request for Anthropic Message Batches API."""
+    requests = []
+    for i, user_input in enumerate(inputs):
+        requests.append({
+            "custom_id": f"request-{i}",
+            "params": {
+                "model": model,
+                "max_tokens": 4096,
+                "system": system_prompt,
+                "messages": [
+                    {"role": "user", "content": user_input}
+                ]
+            }
+        })
+
+    batch = client.messages.batches.create(requests=requests)
+    return batch.id
+```
+
+### Processing Batch Results
+
+```python
+def batch_results_to_training(results_file, inputs, system_prompt=None):
+    """Convert batch API results into training JSONL."""
+    training = []
+    with open(results_file) as f:
+        for line in f:
+            result = json.loads(line)
+            idx = int(result["custom_id"].split("-")[1])
+            messages = []
+            if system_prompt:
+                messages.append({"role": "system", "content": system_prompt})
+            messages.append({"role": "user", "content": inputs[idx]})
+            # Extract assistant response from batch result
+            content = result["response"]["body"]["choices"][0]["message"]["content"]
+            messages.append({"role": "assistant", "content": content})
+            training.append({"messages": messages})
+
+    with open("distilled_training.jsonl", "w") as f:
+        for example in training:
+            f.write(json.dumps(example) + "\n")
+    return len(training)
+```
+
+## Path C: Synthetic Bootstrapping
+
+Generate training data from scratch when you have < 1000 real examples. Use as a starting point, then replace with production data as it accumulates.
+
+### Seed Prompt Strategy
+
+```python
+import openai
+
+client = openai.OpenAI()
+
+def generate_synthetic_examples(task_description, seed_examples, n=500, model="gpt-4o"):
+    """Generate diverse synthetic training examples from seed examples."""
+
+    meta_prompt = f"""You are generating training data for an LLM that will be fine-tuned for:
+{task_description}
+
+Here are {len(seed_examples)} real examples of the desired behavior:
+{json.dumps(seed_examples[:5], indent=2)}
+
+Generate a NEW, diverse example. The input should cover a different scenario than
+the seeds. The output should match the quality and style of the examples above.
+
+Return JSON: {{"input": "...", "output": "..."}}"""
+
+    examples = []
+    for i in range(n):
+        response = client.chat.completions.create(
+            model=model,
+            messages=[{"role": "user", "content": meta_prompt}],
+            response_format={"type": "json_object"},
+            temperature=0.9,  # High temp for diversity
+        )
+        example = json.loads(response.choices[0].message.content)
+        examples.append({
+            "messages": [
+                {"role": "user", "content": example["input"]},
+                {"role": "assistant", "content": example["output"]}
+            ]
+        })
+    return examples
+```
+
+### Diversity Strategies
+- Vary temperature (0.7-1.0) across generation batches
+- Use different frontier models (GPT-4o, Claude, Gemini) to reduce model-specific bias
+- Seed with representative prompts from different categories/difficulty levels
+- Include edge cases and adversarial examples explicitly in seed prompts
+
+## Data Quality Validation
+
+### Schema Validation
+
+```python
+def validate_jsonl(filepath):
+    """Validate JSONL training file format."""
+    errors = []
+    valid = 0
+    with open(filepath) as f:
+        for i, line in enumerate(f, 1):
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError as e:
+                errors.append(f"Line {i}: Invalid JSON — {e}")
+                continue
+
+            if "messages" not in obj:
+                errors.append(f"Line {i}: Missing 'messages' key")
+                continue
+
+            msgs = obj["messages"]
+            if not isinstance(msgs, list) or len(msgs) < 2:
+                errors.append(f"Line {i}: 'messages' must be a list with >= 2 entries")
+                continue
+
+            # Check roles
+            has_user = any(m.get("role") == "user" for m in msgs)
+            has_assistant = any(m.get("role") == "assistant" for m in msgs)
+            if not has_user or not has_assistant:
+                errors.append(f"Line {i}: Must have at least one user and one assistant message")
+                continue
+
+            for j, msg in enumerate(msgs):
+                if "role" not in msg or "content" not in msg:
+                    errors.append(f"Line {i}, message {j}: Missing 'role' or 'content'")
+                elif msg["role"] not in ("system", "user", "assistant"):
+                    errors.append(f"Line {i}, message {j}: Invalid role '{msg['role']}'")
+
+            valid += 1
+
+    return {"valid": valid, "errors": errors, "total": valid + len(errors)}
+```
+
+### Deduplication (MinHash)
+
+```python
+from datasketch import MinHash, MinHashLSH
+
+def deduplicate_dataset(examples, threshold=0.8):
+    """Remove near-duplicate examples using MinHash LSH."""
+    lsh = MinHashLSH(threshold=threshold, num_perm=128)
+    unique = []
+
+    for i, ex in enumerate(examples):
+        # Hash the assistant's response (training target)
+        text = ex["messages"][-1]["content"]
+        m = MinHash(num_perm=128)
+        for word in text.lower().split():
+            m.update(word.encode("utf-8"))
+
+        key = f"doc-{i}"
+        if not lsh.query(m):
+            lsh.insert(key, m)
+            unique.append(ex)
+
+    removed = len(examples) - len(unique)
+    print(f"Removed {removed} duplicates ({removed/len(examples)*100:.1f}%)")
+    return unique
+```
+
+### Diversity Metrics
+
+```python
+from collections import Counter
+
+def distinct_n(texts, n=2):
+    """Calculate distinct-n metric (ratio of unique n-grams to total n-grams)."""
+    total_ngrams = Counter()
+    for text in texts:
+        words = text.lower().split()
+        ngrams = [tuple(words[i:i+n]) for i in range(len(words)-n+1)]
+        total_ngrams.update(ngrams)
+    if sum(total_ngrams.values()) == 0:
+        return 0
+    return len(total_ngrams) / sum(total_ngrams.values())
+
+def dataset_diversity_report(examples):
+    """Generate diversity metrics for a training dataset."""
+    responses = [ex["messages"][-1]["content"] for ex in examples]
+    inputs = [m["content"] for ex in examples for m in ex["messages"] if m["role"] == "user"]
+
+    report = {
+        "total_examples": len(examples),
+        "avg_response_length": sum(len(r.split()) for r in responses) / len(responses),
+        "avg_input_length": sum(len(i.split()) for i in inputs) / len(inputs),
+        "distinct_1": distinct_n(responses, 1),
+        "distinct_2": distinct_n(responses, 2),
+        "distinct_3": distinct_n(responses, 3),
+    }
+    return report
+```
+
+## Train/Eval Split
+
+```python
+import random
+
+def split_dataset(examples, eval_ratio=0.1, production_indices=None):
+    """Split dataset into train/eval, keeping production data in eval for ground truth.
+
+    Args:
+        examples: List of training examples
+        eval_ratio: Fraction of data for evaluation (default 10%)
+        production_indices: Indices of real production examples (always go to eval)
+    """
+    production_indices = set(production_indices or [])
+    synthetic = [ex for i, ex in enumerate(examples) if i not in production_indices]
+    production = [ex for i, ex in enumerate(examples) if i in production_indices]
+
+    # Production data goes to eval (ground truth)
+    eval_set = list(production)
+
+    # Fill remaining eval budget from synthetic
+    remaining_eval = max(0, int(len(examples) * eval_ratio) - len(eval_set))
+    random.shuffle(synthetic)
+    eval_set.extend(synthetic[:remaining_eval])
+    train_set = synthetic[remaining_eval:]
+
+    print(f"Train: {len(train_set)}, Eval: {len(eval_set)} "
+          f"({len(production)} production + {len(eval_set)-len(production)} synthetic)")
+    return train_set, eval_set
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `JSONDecodeError` | Trailing commas or malformed JSON | Run `validate_jsonl()` and fix flagged lines |
+| Tokenizer mismatch | Data tokenized for wrong model | Always use target model's tokenizer for length checks |
+| Training loss doesn't decrease | Data too noisy or contradictory | Filter low-quality examples, check for duplicates |
+| Model repeats training data | Overfitting on small dataset | Add more diverse examples, reduce epochs |
+| Data leakage | Eval examples appear in training | Use `split_dataset()` with `production_indices` |
+| Encoding errors | Non-UTF-8 characters | `text.encode('utf-8', errors='replace').decode('utf-8')` |
+| Examples too long | Exceeds model context | Truncate or split long conversations, check tokenizer limits |
+
+## Quick Start Checklist
+
+1. **Identify data source**: Production logs (A), frontier distillation (B), or synthetic (C)
+2. **Format to JSONL**: Standard chat format with messages array
+3. **Validate**: Run `validate_jsonl()` on the output file
+4. **Deduplicate**: Run MinHash dedup with 0.8 threshold
+5. **Check diversity**: Run `dataset_diversity_report()`, aim for distinct-2 > 0.5
+6. **Split**: 90/10 train/eval, production data in eval set
+7. **Count tokens**: Verify no examples exceed model's context window
+8. **Proceed to training**: Load `tinker` or `unsloth` skill for next step