@synsci/cli-darwin-arm64 1.1.72 → 1.1.74

package/bin/skills/training-data-pipeline/references/data-quality.md

@@ -0,0 +1,136 @@
+# Data Quality Reference
+
+## Overview
+
+Data quality directly determines model quality. This reference covers validation,
+filtering, and quality metrics for training data.
+
+## Quality Dimensions
+
+| Dimension | What It Measures | Target |
+|-----------|-----------------|--------|
+| Correctness | Are responses factually accurate? | Manual review sample |
+| Consistency | Do similar inputs produce similar outputs? | Low variance on paraphrases |
+| Completeness | Are responses thorough? | Task-dependent length targets |
+| Format compliance | Do responses match required format? | 100% schema validation pass |
+| Diversity | Does the dataset cover the input space? | distinct-2 > 0.5 |
+| Deduplication | Are near-duplicates removed? | < 5% duplicate rate |
+
+## Automated Quality Checks
+
+### Token Length Distribution
+
+```python
+from transformers import AutoTokenizer
+
+def token_length_analysis(examples, model_name="meta-llama/Llama-3.1-8B"):
+    """Analyze token lengths to catch outliers and set training params."""
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+    lengths = []
+    for ex in examples:
+        text = tokenizer.apply_chat_template(ex["messages"], tokenize=False)
+        tokens = tokenizer.encode(text)
+        lengths.append(len(tokens))
+
+    import numpy as np
+    lengths = np.array(lengths)
+    return {
+        "count": len(lengths),
+        "mean": float(np.mean(lengths)),
+        "median": float(np.median(lengths)),
+        "p95": float(np.percentile(lengths, 95)),
+        "p99": float(np.percentile(lengths, 99)),
+        "max": int(np.max(lengths)),
+        "recommended_max_seq_length": int(np.percentile(lengths, 99) * 1.1),
+    }
+```
+
+### Response Quality Scoring
+
+Use an LLM judge to score training examples:
+
+```python
+def score_example_quality(example, criteria, model="gpt-4o-mini"):
+    """Score a training example on 1-5 scale using LLM judge."""
+    user_msg = next(m["content"] for m in example["messages"] if m["role"] == "user")
+    assistant_msg = next(m["content"] for m in example["messages"] if m["role"] == "assistant")
+
+    prompt = f"""Rate this response on a 1-5 scale for each criterion.
+
+Input: {user_msg}
+Response: {assistant_msg}
+
+Criteria:
+{criteria}
+
+Return JSON: {{"scores": {{"criterion_name": score, ...}}, "overall": score, "reasoning": "..."}}"""
+
+    # Call LLM and parse response
+    # Filter examples below threshold (e.g., overall < 3)
+```
+
+## PII Detection and Redaction
+
+```python
+import re
+
+PII_PATTERNS = {
+    "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
+    "phone": r'\b(?:\+?1[-.\s]?)?(?:\(?\d{3}\)?[-.\s]?)?\d{3}[-.\s]?\d{4}\b',
+    "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
+    "credit_card": r'\b(?:\d{4}[-\s]?){3}\d{4}\b',
+    "ip_address": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
+}
+
+def redact_pii(text, patterns=PII_PATTERNS):
+    """Replace PII patterns with placeholder tokens."""
+    for name, pattern in patterns.items():
+        text = re.sub(pattern, f"[{name.upper()}_REDACTED]", text)
+    return text
+```
+
+## Dataset Health Report
+
+```python
+def dataset_health_report(filepath):
+    """Generate a comprehensive health report for a training dataset."""
+    import json
+
+    examples = []
+    with open(filepath) as f:
+        for line in f:
+            examples.append(json.loads(line))
+
+    # Basic stats
+    report = {
+        "total_examples": len(examples),
+        "avg_turns_per_example": sum(
+            len(ex["messages"]) for ex in examples
+        ) / len(examples),
+    }
+
+    # Role distribution
+    roles = {}
+    for ex in examples:
+        for msg in ex["messages"]:
+            roles[msg["role"]] = roles.get(msg["role"], 0) + 1
+    report["role_distribution"] = roles
+
+    # Length stats
+    response_lengths = [
+        len(ex["messages"][-1]["content"].split())
+        for ex in examples
+    ]
+    report["response_word_count"] = {
+        "min": min(response_lengths),
+        "max": max(response_lengths),
+        "mean": sum(response_lengths) / len(response_lengths),
+    }
+
+    # Empty/short responses
+    short = sum(1 for l in response_lengths if l < 10)
+    report["short_responses"] = f"{short} ({short/len(examples)*100:.1f}%)"
+
+    return report
+```

package/bin/skills/training-data-pipeline/references/frontier-distillation.md

@@ -0,0 +1,129 @@
+# Frontier Distillation Reference
+
+## Overview
+
+Frontier distillation uses a large teacher model (GPT-4o, Claude, Gemini) to generate
+high-quality labels for your production inputs. The student model learns to replicate
+the teacher's behavior on your specific task at a fraction of the inference cost.
+
+## When to Use
+
+- You have production inputs but no gold-standard labels
+- You want to match frontier quality on a specific task
+- Volume justifies the one-time labeling cost (labels are reusable)
+- Your task is narrow enough that a smaller model can learn it
+
+## Batch API Comparison
+
+| Provider | API | Discount | Turnaround | Max Batch |
+|----------|-----|----------|------------|-----------|
+| OpenAI | Batch API | 50% off | Up to 24h | 50,000 requests |
+| Anthropic | Message Batches | 50% off | Up to 24h | 100,000 requests |
+| Google | Batch Predict | Varies | Hours | Large |
+
+## Distillation Prompt Design
+
+The quality of distilled data depends on the prompt. Be explicit about format, style, and constraints.
+
+```python
+DISTILLATION_SYSTEM_PROMPT = """You are generating training data for a specialized model.
+
+Task: {task_description}
+
+Requirements:
+- Output format: {format_spec}
+- Tone: {tone}
+- Length: {length_constraint}
+- Must include: {required_elements}
+- Must NOT include: {forbidden_elements}
+
+Produce the highest quality response possible. This will be used as a training
+target for a smaller model."""
+```
+
+### Key Principles
+
+1. **Be explicit** — The teacher model should know exactly what format you need
+2. **Include constraints** — Length, format, required sections, forbidden content
+3. **Match production conditions** — Use the same system prompt you use in production
+4. **Verify quality** — Sample and manually review 50-100 examples before using all
+
+## Quality Filtering
+
+Not all teacher outputs are good training data. Filter before training:
+
+```python
+def filter_distilled_data(examples, min_length=50, max_length=4000):
+    """Filter distilled examples by quality heuristics."""
+    filtered = []
+    for ex in examples:
+        response = ex["messages"][-1]["content"]
+
+        # Length check
+        if len(response) < min_length or len(response) > max_length:
+            continue
+
+        # Refusal detection
+        refusal_phrases = [
+            "I cannot", "I'm unable to", "I don't have access",
+            "As an AI", "I'm not able to"
+        ]
+        if any(phrase.lower() in response.lower() for phrase in refusal_phrases):
+            continue
+
+        # Format compliance (customize per task)
+        # if not response.startswith("{"):  # e.g., JSON output expected
+        #     continue
+
+        filtered.append(ex)
+
+    print(f"Kept {len(filtered)}/{len(examples)} ({len(filtered)/len(examples)*100:.1f}%)")
+    return filtered
+```
+
+## Cost Estimation
+
+```python
+def estimate_distillation_cost(num_examples, avg_input_tokens, avg_output_tokens, model="gpt-4o"):
+    """Estimate batch distillation cost."""
+    # Batch API prices (50% of real-time)
+    prices = {
+        "gpt-4o": {"input": 1.25, "output": 5.00},        # per 1M tokens, batch
+        "gpt-4o-mini": {"input": 0.075, "output": 0.30},   # per 1M tokens, batch
+        "claude-sonnet": {"input": 1.50, "output": 7.50},  # per 1M tokens, batch
+    }
+    p = prices.get(model, prices["gpt-4o"])
+
+    input_cost = (num_examples * avg_input_tokens / 1_000_000) * p["input"]
+    output_cost = (num_examples * avg_output_tokens / 1_000_000) * p["output"]
+    total = input_cost + output_cost
+
+    return {
+        "model": model,
+        "examples": num_examples,
+        "input_cost": f"${input_cost:.2f}",
+        "output_cost": f"${output_cost:.2f}",
+        "total_cost": f"${total:.2f}",
+    }
+```
+
+## Multi-Model Distillation
+
+Using multiple teacher models reduces single-model bias:
+
+```python
+def multi_teacher_distillation(inputs, system_prompt, models=None):
+    """Generate labels from multiple teachers and take majority or best."""
+    models = models or ["gpt-4o", "claude-sonnet-4-5-20250929"]
+
+    # Generate labels from each teacher
+    all_labels = {model: generate_labels(inputs, system_prompt, model) for model in models}
+
+    # Strategy 1: Use best model as primary, others for validation
+    primary = all_labels[models[0]]
+
+    # Strategy 2: Use agreement as quality signal
+    # Keep examples where all teachers agree (highest confidence)
+
+    return primary  # Or implement agreement filtering
+```

package/bin/skills/training-data-pipeline/references/production-data-formatting.md

@@ -0,0 +1,126 @@
+# Production Data Formatting Reference
+
+## Overview
+
+Production data is the most valuable training signal for model specialization. This reference covers patterns for extracting, cleaning, and formatting production data from common sources.
+
+## Data Source Patterns
+
+### REST API Logs
+
+Most production systems log API requests and responses. Common formats:
+
+```python
+# Typical API log structure
+log_entry = {
+    "timestamp": "2026-01-15T10:30:00Z",
+    "request_id": "req_abc123",
+    "user_id": "user_456",
+    "endpoint": "/v1/chat/completions",
+    "input": {
+        "model": "gpt-4o",
+        "messages": [...],
+        "temperature": 0.7
+    },
+    "output": {
+        "choices": [{"message": {"content": "..."}}],
+        "usage": {"prompt_tokens": 150, "completion_tokens": 200}
+    },
+    "latency_ms": 1200,
+    "status": 200
+}
+```
+
+**Extraction pattern**: Pull `input.messages` and `output.choices[0].message.content`, format into standard JSONL.
+
+### Database Records
+
+If your product stores LLM interactions in a database:
+
+```sql
+SELECT
+    system_prompt,
+    user_input,
+    COALESCE(corrected_response, model_response) as target_response,
+    user_feedback
+FROM llm_interactions
+WHERE user_feedback != 'rejected'
+    AND created_at > NOW() - INTERVAL '90 days'
+ORDER BY created_at DESC;
+```
+
+**Key**: Always prefer `corrected_response` over raw `model_response` when available.
+
+### Structured Feedback
+
+If users rate or edit model outputs:
+
+| Signal | Quality | Use |
+|--------|---------|-----|
+| User edited output | Highest | Use edited version as training target |
+| Thumbs up / accepted | High | Use original output as training target |
+| Thumbs down / rejected | Medium | Exclude from SFT, use for DPO (rejected example) |
+| No feedback | Low | Use with caution, filter by heuristics |
+
+## Cleaning Pipeline
+
+```python
+def clean_production_data(examples):
+    """Standard cleaning pipeline for production data."""
+    cleaned = []
+    for ex in examples:
+        messages = ex["messages"]
+
+        # Skip empty or trivial examples
+        assistant_msg = next((m for m in messages if m["role"] == "assistant"), None)
+        if not assistant_msg or len(assistant_msg["content"].strip()) < 10:
+            continue
+
+        # Normalize whitespace
+        for msg in messages:
+            msg["content"] = " ".join(msg["content"].split())
+
+        # Remove PII patterns (customize for your domain)
+        for msg in messages:
+            msg["content"] = redact_pii(msg["content"])
+
+        # Skip if user input is too short (likely a test)
+        user_msg = next((m for m in messages if m["role"] == "user"), None)
+        if user_msg and len(user_msg["content"].strip()) < 5:
+            continue
+
+        cleaned.append({"messages": messages})
+
+    return cleaned
+```
+
+## Multi-Turn Conversations
+
+For products with multi-turn interactions, preserve the full conversation:
+
+```python
+def conversation_to_training(conversation):
+    """Convert a multi-turn conversation to training format.
+    Each assistant turn becomes a training example with full history."""
+    examples = []
+    messages = []
+
+    for turn in conversation["turns"]:
+        messages.append({"role": turn["role"], "content": turn["content"]})
+
+        # Create an example at each assistant turn
+        if turn["role"] == "assistant":
+            examples.append({"messages": list(messages)})
+
+    return examples
+```
+
+## Volume Guidelines
+
+| Dataset Size | Expected Quality | Recommended Approach |
+|-------------|-----------------|---------------------|
+| < 100 | Insufficient for SFT | Use synthetic bootstrapping first |
+| 100-1,000 | Minimum viable | LoRA fine-tune, careful eval |
+| 1,000-10,000 | Good | Standard LoRA or QLoRA |
+| 10,000-100,000 | Strong | Full fine-tune viable |
+| > 100,000 | Excellent | Multi-epoch training, curriculum learning |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@synsci/cli-darwin-arm64",
-  "version": "1.1.72",
+  "version": "1.1.74",
   "os": [
     "darwin"
   ],