@synsci/cli-darwin-x64 1.1.49

package/bin/skills/tinker-training-cost/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: tinker-training-cost
+description: Calculates training costs for Tinker fine-tuning jobs. Use when estimating costs for Tinker LLM training, counting tokens in datasets, or comparing Tinker model training prices. Tokenizes datasets using the correct model tokenizer and provides accurate cost estimates.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Tinker, Training Cost, Token Counting, Fine-Tuning, Pricing]
+dependencies: [transformers>=4.40.0]
+---
+
+# Tinker Training Cost Calculator
+
+Calculate training costs for Tinker fine-tuning jobs by tokenizing your dataset with the correct model tokenizer and applying current pricing.
+
+## Quick Start
+
+Use the bundled script to calculate training costs:
+
+```bash
+# List available models and pricing
+python scripts/calculate_cost.py --list-models
+
+# Calculate cost for a JSONL dataset
+python scripts/calculate_cost.py training_data.jsonl --model Qwen3-8B --epochs 3
+
+# Output as JSON
+python scripts/calculate_cost.py training_data.jsonl --model Llama-3.1-70B --json
+```
+
+The script:
+1. Loads the correct tokenizer for the selected model
+2. Counts tokens in your JSONL file (supports chat, text, and instruction formats)
+3. Calculates the estimated training cost
+
+## Cost Formula
+
+```
+Training Cost = (total_tokens × epochs × train_price_per_million) / 1_000_000
+```
+
+Where:
+- `total_tokens` = tokens in your training dataset (from tokenization)
+- `epochs` = number of training passes (default: 3)
+- `train_price_per_million` = model-specific training rate from pricing table
+
+---
+
+## Tinker Pricing
+
+> **All prices as of January 5, 2026**
+> Source: https://thinkingmachines.ai/tinker/
+
+All prices are in **USD per million tokens**.
+
+| Category | Description |
+|----------|-------------|
+| **Prefill** | Processing input context (inference) |
+| **Sample** | Generating output tokens (inference) |
+| **Train** | Training/fine-tuning tokens |
+
+### Qwen Models
+
+| Model | Prefill | Sample | Train |
+|-------|---------|--------|-------|
+| Qwen3-4B-Instruct-2507 | $0.07 | $0.22 | $0.22 |
+| Qwen3-8B | $0.13 | $0.40 | $0.40 |
+| Qwen3-30B-A3B | $0.12 | $0.30 | $0.36 |
+| Qwen3-VL-30B-A3B-Instruct | $0.18 | $0.44 | $0.53 |
+| Qwen3-32B | $0.49 | $1.47 | $1.47 |
+| Qwen3-235B-Instruct-2507 | $0.68 | $1.70 | $2.04 |
+| Qwen3-VL-235B-A22B-Instruct | $1.02 | $2.56 | $3.07 |
+
+### Llama Models
+
+| Model | Prefill | Sample | Train |
+|-------|---------|--------|-------|
+| Llama-3.2-1B | $0.03 | $0.09 | $0.09 |
+| Llama-3.2-3B | $0.06 | $0.18 | $0.18 |
+| Llama-3.1-8B | $0.13 | $0.40 | $0.40 |
+| Llama-3.1-70B | $1.05 | $3.16 | $3.16 |
+
+### DeepSeek Models
+
+| Model | Prefill | Sample | Train |
+|-------|---------|--------|-------|
+| DeepSeek-V3.1 | $1.13 | $2.81 | $3.38 |
+
+### GPT-OSS Models
+
+| Model | Prefill | Sample | Train |
+|-------|---------|--------|-------|
+| GPT-OSS-120B | $0.18 | $0.44 | $0.52 |
+| GPT-OSS-20B | $0.12 | $0.30 | $0.36 |
+
+### Moonshot Models
+
+| Model | Prefill | Sample | Train |
+|-------|---------|--------|-------|
+| Kimi-K2-Thinking | $0.98 | $2.44 | $2.93 |
+
+---
+
+## Model-to-Tokenizer Mapping
+
+Use the correct HuggingFace tokenizer for accurate token counting:
+
+| Model | HuggingFace Tokenizer |
+|-------|----------------------|
+| Qwen3-4B-Instruct-2507 | `Qwen/Qwen3-4B` |
+| Qwen3-8B | `Qwen/Qwen3-8B` |
+| Qwen3-30B-A3B | `Qwen/Qwen3-30B-A3B` |
+| Qwen3-32B | `Qwen/Qwen3-32B` |
+| Qwen3-235B-Instruct-2507 | `Qwen/Qwen3-235B-A22B-Instruct` |
+| Qwen3-VL-* | `Qwen/Qwen2.5-VL-7B-Instruct` (shared VL tokenizer) |
+| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` |
+| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` |
+| Llama-3.1-8B | `meta-llama/Llama-3.1-8B-Instruct` |
+| Llama-3.1-70B | `meta-llama/Llama-3.1-70B-Instruct` |
+| DeepSeek-V3.1 | `deepseek-ai/DeepSeek-V3` |
+| GPT-OSS-* | `Qwen/Qwen3-8B` (compatible tokenizer) |
+| Kimi-K2-Thinking | `moonshotai/Kimi-K2-Instruct` |
+
+---
+
+## Tokenization
+
+The bundled `scripts/calculate_cost.py` handles tokenization automatically. For custom use:
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B", trust_remote_code=True)
+token_count = len(tokenizer.encode("Your training text here"))
+```
+
+### Supported JSONL Formats
+
+The script handles these training data formats:
+
+**Chat format** (recommended):
+```json
+{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
+```
+
+**Text format**:
+```json
+{"text": "Your training text here"}
+```
+
+**Instruction format** (Alpaca-style):
+```json
+{"instruction": "...", "input": "...", "output": "..."}
+```
+
+---
+
+## Quick Cost Examples
+
+### Example 1: Qwen3-8B on 1M tokens, 3 epochs
+```
+Dataset tokens: 1,000,000
+Training tokens: 1,000,000 × 3 = 3,000,000
+Cost: 3.0M × $0.40/M = $1.20
+```
+
+### Example 2: Llama-3.1-70B on 5M tokens, 2 epochs
+```
+Dataset tokens: 5,000,000
+Training tokens: 5,000,000 × 2 = 10,000,000
+Cost: 10.0M × $3.16/M = $31.60
+```
+
+### Example 3: Qwen3-235B on 2M tokens, 4 epochs
+```
+Dataset tokens: 2,000,000
+Training tokens: 2,000,000 × 4 = 8,000,000
+Cost: 8.0M × $2.04/M = $16.32
+```
+
+---
+
+## Important Notes
+
+1. **LoRA Fine-Tuning**: Tinker uses Low-Rank Adaptation (LoRA), not full fine-tuning
+2. **Token Counting**: Always use the model's native tokenizer for accurate counts - different tokenizers produce different token counts for the same text
+3. **Vision Models**: VL models have higher costs due to image processing overhead
+4. **trust_remote_code**: Required for some tokenizers (Qwen, DeepSeek)

package/bin/skills/tinker-training-cost/scripts/calculate_cost.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""Calculate training costs for Tinker fine-tuning jobs."""
+
+import argparse
+import json
+import sys
+
+MODELS = {
+    "Qwen3-4B-Instruct-2507": {"tokenizer": "Qwen/Qwen3-4B", "prefill": 0.07, "sample": 0.22, "train": 0.22},
+    "Qwen3-8B": {"tokenizer": "Qwen/Qwen3-8B", "prefill": 0.13, "sample": 0.40, "train": 0.40},
+    "Qwen3-30B-A3B": {"tokenizer": "Qwen/Qwen3-30B-A3B", "prefill": 0.12, "sample": 0.30, "train": 0.36},
+    "Qwen3-VL-30B-A3B-Instruct": {"tokenizer": "Qwen/Qwen2.5-VL-7B-Instruct", "prefill": 0.18, "sample": 0.44, "train": 0.53},
+    "Qwen3-32B": {"tokenizer": "Qwen/Qwen3-32B", "prefill": 0.49, "sample": 1.47, "train": 1.47},
+    "Qwen3-235B-Instruct-2507": {"tokenizer": "Qwen/Qwen3-235B-A22B-Instruct", "prefill": 0.68, "sample": 1.70, "train": 2.04},
+    "Qwen3-VL-235B-A22B-Instruct": {"tokenizer": "Qwen/Qwen2.5-VL-7B-Instruct", "prefill": 1.02, "sample": 2.56, "train": 3.07},
+    "Llama-3.2-1B": {"tokenizer": "meta-llama/Llama-3.2-1B-Instruct", "prefill": 0.03, "sample": 0.09, "train": 0.09},
+    "Llama-3.2-3B": {"tokenizer": "meta-llama/Llama-3.2-3B-Instruct", "prefill": 0.06, "sample": 0.18, "train": 0.18},
+    "Llama-3.1-8B": {"tokenizer": "meta-llama/Llama-3.1-8B-Instruct", "prefill": 0.13, "sample": 0.40, "train": 0.40},
+    "Llama-3.1-70B": {"tokenizer": "meta-llama/Llama-3.1-70B-Instruct", "prefill": 1.05, "sample": 3.16, "train": 3.16},
+    "DeepSeek-V3.1": {"tokenizer": "deepseek-ai/DeepSeek-V3", "prefill": 1.13, "sample": 2.81, "train": 3.38},
+    "GPT-OSS-120B": {"tokenizer": "Qwen/Qwen3-8B", "prefill": 0.18, "sample": 0.44, "train": 0.52},
+    "GPT-OSS-20B": {"tokenizer": "Qwen/Qwen3-8B", "prefill": 0.12, "sample": 0.30, "train": 0.36},
+    "Kimi-K2-Thinking": {"tokenizer": "moonshotai/Kimi-K2-Instruct", "prefill": 0.98, "sample": 2.44, "train": 2.93},
+}
+
+
+def extract_text(row):
+    if "messages" in row:
+        return " ".join(m.get("content", "") for m in row["messages"])
+    if "text" in row:
+        return row["text"]
+    if "instruction" in row:
+        parts = [row.get("instruction", ""), row.get("input", ""), row.get("output", "")]
+        return " ".join(p for p in parts if p)
+    return ""
+
+
+def count_tokens(file_path, tokenizer):
+    total = 0
+    lines = 0
+    with open(file_path) as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            row = json.loads(line)
+            text = extract_text(row)
+            total += len(tokenizer.encode(text))
+            lines += 1
+    return total, lines
+
+
+def list_models():
+    print(f"{'Model':<35} {'Train $/M':>10} {'Prefill $/M':>12} {'Sample $/M':>11}")
+    print("-" * 70)
+    for name, info in MODELS.items():
+        print(f"{name:<35} ${info['train']:>8.2f} ${info['prefill']:>10.2f} ${info['sample']:>9.2f}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Calculate Tinker training costs")
+    parser.add_argument("file", nargs="?", help="JSONL training data file")
+    parser.add_argument("--model", "-m", default="Qwen3-8B", help="Model name (default: Qwen3-8B)")
+    parser.add_argument("--epochs", "-e", type=int, default=3, help="Number of epochs (default: 3)")
+    parser.add_argument("--list-models", action="store_true", help="List available models and pricing")
+    parser.add_argument("--json", action="store_true", help="Output as JSON")
+    args = parser.parse_args()
+
+    if args.list_models:
+        list_models()
+        return
+
+    if not args.file:
+        parser.error("Please provide a JSONL file or use --list-models")
+
+    if args.model not in MODELS:
+        print(f"Unknown model: {args.model}", file=sys.stderr)
+        print(f"Available: {', '.join(MODELS.keys())}", file=sys.stderr)
+        sys.exit(1)
+
+    model_info = MODELS[args.model]
+
+    try:
+        from transformers import AutoTokenizer
+    except ImportError:
+        print("pip install transformers to use tokenization", file=sys.stderr)
+        sys.exit(1)
+
+    print(f"Loading tokenizer: {model_info['tokenizer']}...", file=sys.stderr)
+    tokenizer = AutoTokenizer.from_pretrained(model_info["tokenizer"], trust_remote_code=True)
+
+    print(f"Counting tokens in {args.file}...", file=sys.stderr)
+    total_tokens, num_examples = count_tokens(args.file, tokenizer)
+
+    training_tokens = total_tokens * args.epochs
+    cost = (training_tokens * model_info["train"]) / 1_000_000
+
+    result = {
+        "model": args.model,
+        "tokenizer": model_info["tokenizer"],
+        "file": args.file,
+        "examples": num_examples,
+        "dataset_tokens": total_tokens,
+        "epochs": args.epochs,
+        "training_tokens": training_tokens,
+        "train_price_per_million": model_info["train"],
+        "estimated_cost_usd": round(cost, 4),
+    }
+
+    if args.json:
+        print(json.dumps(result, indent=2))
+    else:
+        print(f"\nModel:            {args.model}")
+        print(f"Examples:         {num_examples:,}")
+        print(f"Dataset tokens:   {total_tokens:,}")
+        print(f"Epochs:           {args.epochs}")
+        print(f"Training tokens:  {training_tokens:,}")
+        print(f"Price:            ${model_info['train']}/M tokens")
+        print(f"Estimated cost:   ${cost:.4f}")
+
+
+if __name__ == "__main__":
+    main()