@synsci/cli-darwin-x64-baseline 1.1.73 → 1.1.75

package/bin/skills/training-data-pipeline/SKILL.md

@@ -0,0 +1,427 @@
+---
+name: training-data-pipeline
+description: Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train/eval splitting.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Training Data, Data Pipeline, Fine-Tuning, Distillation, Synthetic Data, Production Data, JSONL, Data Quality]
+dependencies: [datasets, transformers, openai]
+---
+
+# Training Data Pipeline
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- **Format production logs** into SFT training data (API logs, user corrections, accept/reject signals)
+- **Distill from frontier models** using batch APIs (OpenAI, Anthropic) to label production inputs
+- **Bootstrap synthetic data** when fewer than 1000 real examples exist
+- **Validate data quality** before training (dedup, schema check, diversity metrics)
+- **Split data** into train/eval sets with production data reserved for evaluation
+
+### Three Data Paths
+
+| Path | When to Use | Data Source | Cost |
+|------|-------------|-------------|------|
+| A) Production data | Have API logs or user feedback | Your own production systems | Free (already collected) |
+| B) Frontier distillation | Have production inputs but no labels | OpenAI/Anthropic batch APIs | ~50% of real-time API cost |
+| C) Synthetic bootstrap | < 1000 real examples | Frontier model generation | Varies by volume |
+
+**Always prefer Path A** — production data is the moat competitors can't replicate.
+
+## JSONL Chat Format
+
+All training platforms (Tinker, Unsloth, TRL, Axolotl) accept this standard chat format:
+
+```jsonl
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]}
+{"messages": [{"role": "user", "content": "Translate to French: Hello"}, {"role": "assistant", "content": "Bonjour"}]}
+```
+
+### Format Rules
+- One JSON object per line, no trailing commas
+- `messages` array with `role` and `content` fields
+- Roles: `system` (optional, first only), `user`, `assistant` (alternating)
+- Multi-turn: alternate user/assistant pairs within a single messages array
+- UTF-8 encoding, no BOM
+- `assistant` messages are the training targets — everything else is context
+
+### Platform-Specific Notes
+
+**Tinker**: Standard chat format above. Max 32K tokens per example. System message optional.
+
+**Unsloth/TRL**: Same format. Also accepts `{"prompt": "...", "completion": "..."}` for simple pairs. Chat format preferred for multi-turn.
+
+**Axolotl**: Supports multiple formats via config. Recommend `chat_template` type with standard JSONL.
+
+## Path A: Production Data Collection
+
+### From API Logs
+
+If you log API requests/responses, convert them directly:
+
+```python
+import json
+
+def api_log_to_training(log_entry):
+    """Convert an API request/response log to training format."""
+    messages = []
+
+    # Add system prompt if present
+    if log_entry.get("system_prompt"):
+        messages.append({
+            "role": "system",
+            "content": log_entry["system_prompt"]
+        })
+
+    # Add the user's input
+    messages.append({
+        "role": "user",
+        "content": log_entry["user_input"]
+    })
+
+    # Add the response (use corrected version if available)
+    response = log_entry.get("corrected_response") or log_entry["api_response"]
+    messages.append({
+        "role": "assistant",
+        "content": response
+    })
+
+    return {"messages": messages}
+
+# Process logs
+with open("api_logs.jsonl") as f, open("training_data.jsonl", "w") as out:
+    for line in f:
+        log = json.loads(line)
+        example = api_log_to_training(log)
+        out.write(json.dumps(example) + "\n")
+```
+
+### From User Corrections
+
+User corrections (edits to model output) are the highest-quality training signal:
+
+```python
+def correction_to_training(original_input, corrected_output, system_prompt=None):
+    """Convert a user correction into a training example.
+    The corrected output becomes the training target."""
+    messages = []
+    if system_prompt:
+        messages.append({"role": "system", "content": system_prompt})
+    messages.append({"role": "user", "content": original_input})
+    messages.append({"role": "assistant", "content": corrected_output})
+    return {"messages": messages}
+```
+
+### From Accept/Reject Signals
+
+If users accept or reject model outputs, use accepted outputs as positive examples:
+
+```python
+def filter_accepted(logs):
+    """Keep only examples where user accepted the output."""
+    accepted = []
+    for log in logs:
+        if log.get("user_action") == "accepted":
+            accepted.append({
+                "messages": [
+                    {"role": "user", "content": log["input"]},
+                    {"role": "assistant", "content": log["output"]}
+                ]
+            })
+    return accepted
+```
+
+## Path B: Frontier Distillation
+
+Use frontier models to label your production inputs. Best when you have real inputs but no gold labels.
+
+### OpenAI Batch API (50% discount)
+
+```python
+import json
+
+def create_batch_file(inputs, system_prompt, model="gpt-4o"):
+    """Create a batch file for OpenAI Batch API."""
+    requests = []
+    for i, user_input in enumerate(inputs):
+        requests.append({
+            "custom_id": f"request-{i}",
+            "method": "POST",
+            "url": "/v1/chat/completions",
+            "body": {
+                "model": model,
+                "messages": [
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_input}
+                ],
+                "max_tokens": 4096
+            }
+        })
+
+    with open("batch_input.jsonl", "w") as f:
+        for req in requests:
+            f.write(json.dumps(req) + "\n")
+    return "batch_input.jsonl"
+
+# Submit batch
+# openai api batches create -i batch_input.jsonl -e /v1/chat/completions -c 24h
+```
+
+### Anthropic Batch API
+
+```python
+import anthropic
+
+client = anthropic.Anthropic()
+
+def create_anthropic_batch(inputs, system_prompt, model="claude-sonnet-4-5-20250929"):
+    """Create batch request for Anthropic Message Batches API."""
+    requests = []
+    for i, user_input in enumerate(inputs):
+        requests.append({
+            "custom_id": f"request-{i}",
+            "params": {
+                "model": model,
+                "max_tokens": 4096,
+                "system": system_prompt,
+                "messages": [
+                    {"role": "user", "content": user_input}
+                ]
+            }
+        })
+
+    batch = client.messages.batches.create(requests=requests)
+    return batch.id
+```
+
+### Processing Batch Results
+
+```python
+def batch_results_to_training(results_file, inputs, system_prompt=None):
+    """Convert batch API results into training JSONL."""
+    training = []
+    with open(results_file) as f:
+        for line in f:
+            result = json.loads(line)
+            idx = int(result["custom_id"].split("-")[1])
+            messages = []
+            if system_prompt:
+                messages.append({"role": "system", "content": system_prompt})
+            messages.append({"role": "user", "content": inputs[idx]})
+            # Extract assistant response from batch result
+            content = result["response"]["body"]["choices"][0]["message"]["content"]
+            messages.append({"role": "assistant", "content": content})
+            training.append({"messages": messages})
+
+    with open("distilled_training.jsonl", "w") as f:
+        for example in training:
+            f.write(json.dumps(example) + "\n")
+    return len(training)
+```
+
+## Path C: Synthetic Bootstrapping
+
+Generate training data from scratch when you have < 1000 real examples. Use as a starting point, then replace with production data as it accumulates.
+
+### Seed Prompt Strategy
+
+```python
+import openai
+
+client = openai.OpenAI()
+
+def generate_synthetic_examples(task_description, seed_examples, n=500, model="gpt-4o"):
+    """Generate diverse synthetic training examples from seed examples."""
+
+    meta_prompt = f"""You are generating training data for an LLM that will be fine-tuned for:
+{task_description}
+
+Here are {len(seed_examples)} real examples of the desired behavior:
+{json.dumps(seed_examples[:5], indent=2)}
+
+Generate a NEW, diverse example. The input should cover a different scenario than
+the seeds. The output should match the quality and style of the examples above.
+
+Return JSON: {{"input": "...", "output": "..."}}"""
+
+    examples = []
+    for i in range(n):
+        response = client.chat.completions.create(
+            model=model,
+            messages=[{"role": "user", "content": meta_prompt}],
+            response_format={"type": "json_object"},
+            temperature=0.9,  # High temp for diversity
+        )
+        example = json.loads(response.choices[0].message.content)
+        examples.append({
+            "messages": [
+                {"role": "user", "content": example["input"]},
+                {"role": "assistant", "content": example["output"]}
+            ]
+        })
+    return examples
+```
+
+### Diversity Strategies
+- Vary temperature (0.7-1.0) across generation batches
+- Use different frontier models (GPT-4o, Claude, Gemini) to reduce model-specific bias
+- Seed with representative prompts from different categories/difficulty levels
+- Include edge cases and adversarial examples explicitly in seed prompts
+
+## Data Quality Validation
+
+### Schema Validation
+
+```python
+def validate_jsonl(filepath):
+    """Validate JSONL training file format."""
+    errors = []
+    valid = 0
+    with open(filepath) as f:
+        for i, line in enumerate(f, 1):
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError as e:
+                errors.append(f"Line {i}: Invalid JSON — {e}")
+                continue
+
+            if "messages" not in obj:
+                errors.append(f"Line {i}: Missing 'messages' key")
+                continue
+
+            msgs = obj["messages"]
+            if not isinstance(msgs, list) or len(msgs) < 2:
+                errors.append(f"Line {i}: 'messages' must be a list with >= 2 entries")
+                continue
+
+            # Check roles
+            has_user = any(m.get("role") == "user" for m in msgs)
+            has_assistant = any(m.get("role") == "assistant" for m in msgs)
+            if not has_user or not has_assistant:
+                errors.append(f"Line {i}: Must have at least one user and one assistant message")
+                continue
+
+            for j, msg in enumerate(msgs):
+                if "role" not in msg or "content" not in msg:
+                    errors.append(f"Line {i}, message {j}: Missing 'role' or 'content'")
+                elif msg["role"] not in ("system", "user", "assistant"):
+                    errors.append(f"Line {i}, message {j}: Invalid role '{msg['role']}'")
+
+            valid += 1
+
+    return {"valid": valid, "errors": errors, "total": valid + len(errors)}
+```
+
+### Deduplication (MinHash)
+
+```python
+from datasketch import MinHash, MinHashLSH
+
+def deduplicate_dataset(examples, threshold=0.8):
+    """Remove near-duplicate examples using MinHash LSH."""
+    lsh = MinHashLSH(threshold=threshold, num_perm=128)
+    unique = []
+
+    for i, ex in enumerate(examples):
+        # Hash the assistant's response (training target)
+        text = ex["messages"][-1]["content"]
+        m = MinHash(num_perm=128)
+        for word in text.lower().split():
+            m.update(word.encode("utf-8"))
+
+        key = f"doc-{i}"
+        if not lsh.query(m):
+            lsh.insert(key, m)
+            unique.append(ex)
+
+    removed = len(examples) - len(unique)
+    print(f"Removed {removed} duplicates ({removed/len(examples)*100:.1f}%)")
+    return unique
+```
+
+### Diversity Metrics
+
+```python
+from collections import Counter
+
+def distinct_n(texts, n=2):
+    """Calculate distinct-n metric (ratio of unique n-grams to total n-grams)."""
+    total_ngrams = Counter()
+    for text in texts:
+        words = text.lower().split()
+        ngrams = [tuple(words[i:i+n]) for i in range(len(words)-n+1)]
+        total_ngrams.update(ngrams)
+    if sum(total_ngrams.values()) == 0:
+        return 0
+    return len(total_ngrams) / sum(total_ngrams.values())
+
+def dataset_diversity_report(examples):
+    """Generate diversity metrics for a training dataset."""
+    responses = [ex["messages"][-1]["content"] for ex in examples]
+    inputs = [m["content"] for ex in examples for m in ex["messages"] if m["role"] == "user"]
+
+    report = {
+        "total_examples": len(examples),
+        "avg_response_length": sum(len(r.split()) for r in responses) / len(responses),
+        "avg_input_length": sum(len(i.split()) for i in inputs) / len(inputs),
+        "distinct_1": distinct_n(responses, 1),
+        "distinct_2": distinct_n(responses, 2),
+        "distinct_3": distinct_n(responses, 3),
+    }
+    return report
+```
+
+## Train/Eval Split
+
+```python
+import random
+
+def split_dataset(examples, eval_ratio=0.1, production_indices=None):
+    """Split dataset into train/eval, keeping production data in eval for ground truth.
+
+    Args:
+        examples: List of training examples
+        eval_ratio: Fraction of data for evaluation (default 10%)
+        production_indices: Indices of real production examples (always go to eval)
+    """
+    production_indices = set(production_indices or [])
+    synthetic = [ex for i, ex in enumerate(examples) if i not in production_indices]
+    production = [ex for i, ex in enumerate(examples) if i in production_indices]
+
+    # Production data goes to eval (ground truth)
+    eval_set = list(production)
+
+    # Fill remaining eval budget from synthetic
+    remaining_eval = max(0, int(len(examples) * eval_ratio) - len(eval_set))
+    random.shuffle(synthetic)
+    eval_set.extend(synthetic[:remaining_eval])
+    train_set = synthetic[remaining_eval:]
+
+    print(f"Train: {len(train_set)}, Eval: {len(eval_set)} "
+          f"({len(production)} production + {len(eval_set)-len(production)} synthetic)")
+    return train_set, eval_set
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `JSONDecodeError` | Trailing commas or malformed JSON | Run `validate_jsonl()` and fix flagged lines |
+| Tokenizer mismatch | Data tokenized for wrong model | Always use target model's tokenizer for length checks |
+| Training loss doesn't decrease | Data too noisy or contradictory | Filter low-quality examples, check for duplicates |
+| Model repeats training data | Overfitting on small dataset | Add more diverse examples, reduce epochs |
+| Data leakage | Eval examples appear in training | Use `split_dataset()` with `production_indices` |
+| Encoding errors | Non-UTF-8 characters | `text.encode('utf-8', errors='replace').decode('utf-8')` |
+| Examples too long | Exceeds model context | Truncate or split long conversations, check tokenizer limits |
+
+## Quick Start Checklist
+
+1. **Identify data source**: Production logs (A), frontier distillation (B), or synthetic (C)
+2. **Format to JSONL**: Standard chat format with messages array
+3. **Validate**: Run `validate_jsonl()` on the output file
+4. **Deduplicate**: Run MinHash dedup with 0.8 threshold
+5. **Check diversity**: Run `dataset_diversity_report()`, aim for distinct-2 > 0.5
+6. **Split**: 90/10 train/eval, production data in eval set
+7. **Count tokens**: Verify no examples exceed model's context window
+8. **Proceed to training**: Load `tinker` or `unsloth` skill for next step

package/bin/skills/training-data-pipeline/references/data-quality.md

@@ -0,0 +1,136 @@
+# Data Quality Reference
+
+## Overview
+
+Data quality directly determines model quality. This reference covers validation,
+filtering, and quality metrics for training data.
+
+## Quality Dimensions
+
+| Dimension | What It Measures | Target |
+|-----------|-----------------|--------|
+| Correctness | Are responses factually accurate? | Manual review sample |
+| Consistency | Do similar inputs produce similar outputs? | Low variance on paraphrases |
+| Completeness | Are responses thorough? | Task-dependent length targets |
+| Format compliance | Do responses match required format? | 100% schema validation pass |
+| Diversity | Does the dataset cover the input space? | distinct-2 > 0.5 |
+| Deduplication | Are near-duplicates removed? | < 5% duplicate rate |
+
+## Automated Quality Checks
+
+### Token Length Distribution
+
+```python
+from transformers import AutoTokenizer
+
+def token_length_analysis(examples, model_name="meta-llama/Llama-3.1-8B"):
+    """Analyze token lengths to catch outliers and set training params."""
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+    lengths = []
+    for ex in examples:
+        text = tokenizer.apply_chat_template(ex["messages"], tokenize=False)
+        tokens = tokenizer.encode(text)
+        lengths.append(len(tokens))
+
+    import numpy as np
+    lengths = np.array(lengths)
+    return {
+        "count": len(lengths),
+        "mean": float(np.mean(lengths)),
+        "median": float(np.median(lengths)),
+        "p95": float(np.percentile(lengths, 95)),
+        "p99": float(np.percentile(lengths, 99)),
+        "max": int(np.max(lengths)),
+        "recommended_max_seq_length": int(np.percentile(lengths, 99) * 1.1),
+    }
+```
+
+### Response Quality Scoring
+
+Use an LLM judge to score training examples:
+
+```python
+def score_example_quality(example, criteria, model="gpt-4o-mini"):
+    """Score a training example on 1-5 scale using LLM judge."""
+    user_msg = next(m["content"] for m in example["messages"] if m["role"] == "user")
+    assistant_msg = next(m["content"] for m in example["messages"] if m["role"] == "assistant")
+
+    prompt = f"""Rate this response on a 1-5 scale for each criterion.
+
+Input: {user_msg}
+Response: {assistant_msg}
+
+Criteria:
+{criteria}
+
+Return JSON: {{"scores": {{"criterion_name": score, ...}}, "overall": score, "reasoning": "..."}}"""
+
+    # Call LLM and parse response
+    # Filter examples below threshold (e.g., overall < 3)
+```
+
+## PII Detection and Redaction
+
+```python
+import re
+
+PII_PATTERNS = {
+    "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
+    "phone": r'\b(?:\+?1[-.\s]?)?(?:\(?\d{3}\)?[-.\s]?)?\d{3}[-.\s]?\d{4}\b',
+    "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
+    "credit_card": r'\b(?:\d{4}[-\s]?){3}\d{4}\b',
+    "ip_address": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
+}
+
+def redact_pii(text, patterns=PII_PATTERNS):
+    """Replace PII patterns with placeholder tokens."""
+    for name, pattern in patterns.items():
+        text = re.sub(pattern, f"[{name.upper()}_REDACTED]", text)
+    return text
+```
+
+## Dataset Health Report
+
+```python
+def dataset_health_report(filepath):
+    """Generate a comprehensive health report for a training dataset."""
+    import json
+
+    examples = []
+    with open(filepath) as f:
+        for line in f:
+            examples.append(json.loads(line))
+
+    # Basic stats
+    report = {
+        "total_examples": len(examples),
+        "avg_turns_per_example": sum(
+            len(ex["messages"]) for ex in examples
+        ) / len(examples),
+    }
+
+    # Role distribution
+    roles = {}
+    for ex in examples:
+        for msg in ex["messages"]:
+            roles[msg["role"]] = roles.get(msg["role"], 0) + 1
+    report["role_distribution"] = roles
+
+    # Length stats
+    response_lengths = [
+        len(ex["messages"][-1]["content"].split())
+        for ex in examples
+    ]
+    report["response_word_count"] = {
+        "min": min(response_lengths),
+        "max": max(response_lengths),
+        "mean": sum(response_lengths) / len(response_lengths),
+    }
+
+    # Empty/short responses
+    short = sum(1 for l in response_lengths if l < 10)
+    report["short_responses"] = f"{short} ({short/len(examples)*100:.1f}%)"
+
+    return report
+```

package/bin/skills/training-data-pipeline/references/frontier-distillation.md

@@ -0,0 +1,129 @@
+# Frontier Distillation Reference
+
+## Overview
+
+Frontier distillation uses a large teacher model (GPT-4o, Claude, Gemini) to generate
+high-quality labels for your production inputs. The student model learns to replicate
+the teacher's behavior on your specific task at a fraction of the inference cost.
+
+## When to Use
+
+- You have production inputs but no gold-standard labels
+- You want to match frontier quality on a specific task
+- Volume justifies the one-time labeling cost (labels are reusable)
+- Your task is narrow enough that a smaller model can learn it
+
+## Batch API Comparison
+
+| Provider | API | Discount | Turnaround | Max Batch |
+|----------|-----|----------|------------|-----------|
+| OpenAI | Batch API | 50% off | Up to 24h | 50,000 requests |
+| Anthropic | Message Batches | 50% off | Up to 24h | 100,000 requests |
+| Google | Batch Predict | Varies | Hours | Large |
+
+## Distillation Prompt Design
+
+The quality of distilled data depends on the prompt. Be explicit about format, style, and constraints.
+
+```python
+DISTILLATION_SYSTEM_PROMPT = """You are generating training data for a specialized model.
+
+Task: {task_description}
+
+Requirements:
+- Output format: {format_spec}
+- Tone: {tone}
+- Length: {length_constraint}
+- Must include: {required_elements}
+- Must NOT include: {forbidden_elements}
+
+Produce the highest quality response possible. This will be used as a training
+target for a smaller model."""
+```
+
+### Key Principles
+
+1. **Be explicit** — The teacher model should know exactly what format you need
+2. **Include constraints** — Length, format, required sections, forbidden content
+3. **Match production conditions** — Use the same system prompt you use in production
+4. **Verify quality** — Sample and manually review 50-100 examples before using all
+
+## Quality Filtering
+
+Not all teacher outputs are good training data. Filter before training:
+
+```python
+def filter_distilled_data(examples, min_length=50, max_length=4000):
+    """Filter distilled examples by quality heuristics."""
+    filtered = []
+    for ex in examples:
+        response = ex["messages"][-1]["content"]
+
+        # Length check
+        if len(response) < min_length or len(response) > max_length:
+            continue
+
+        # Refusal detection
+        refusal_phrases = [
+            "I cannot", "I'm unable to", "I don't have access",
+            "As an AI", "I'm not able to"
+        ]
+        if any(phrase.lower() in response.lower() for phrase in refusal_phrases):
+            continue
+
+        # Format compliance (customize per task)
+        # if not response.startswith("{"):  # e.g., JSON output expected
+        #     continue
+
+        filtered.append(ex)
+
+    print(f"Kept {len(filtered)}/{len(examples)} ({len(filtered)/len(examples)*100:.1f}%)")
+    return filtered
+```
+
+## Cost Estimation
+
+```python
+def estimate_distillation_cost(num_examples, avg_input_tokens, avg_output_tokens, model="gpt-4o"):
+    """Estimate batch distillation cost."""
+    # Batch API prices (50% of real-time)
+    prices = {
+        "gpt-4o": {"input": 1.25, "output": 5.00},        # per 1M tokens, batch
+        "gpt-4o-mini": {"input": 0.075, "output": 0.30},   # per 1M tokens, batch
+        "claude-sonnet": {"input": 1.50, "output": 7.50},  # per 1M tokens, batch
+    }
+    p = prices.get(model, prices["gpt-4o"])
+
+    input_cost = (num_examples * avg_input_tokens / 1_000_000) * p["input"]
+    output_cost = (num_examples * avg_output_tokens / 1_000_000) * p["output"]
+    total = input_cost + output_cost
+
+    return {
+        "model": model,
+        "examples": num_examples,
+        "input_cost": f"${input_cost:.2f}",
+        "output_cost": f"${output_cost:.2f}",
+        "total_cost": f"${total:.2f}",
+    }
+```
+
+## Multi-Model Distillation
+
+Using multiple teacher models reduces single-model bias:
+
+```python
+def multi_teacher_distillation(inputs, system_prompt, models=None):
+    """Generate labels from multiple teachers and take majority or best."""
+    models = models or ["gpt-4o", "claude-sonnet-4-5-20250929"]
+
+    # Generate labels from each teacher
+    all_labels = {model: generate_labels(inputs, system_prompt, model) for model in models}
+
+    # Strategy 1: Use best model as primary, others for validation
+    primary = all_labels[models[0]]
+
+    # Strategy 2: Use agreement as quality signal
+    # Keep examples where all teachers agree (highest confidence)
+
+    return primary  # Or implement agreement filtering
+```