@synsci/cli-darwin-x64 1.1.49

package/bin/skills/tinker/SKILL.md

@@ -0,0 +1,362 @@
+---
+name: tinker-fine-tuning
+description: Provides guidance for fine-tuning LLMs using the Tinker cloud training API from Thinking Machines Lab. Use when running supervised fine-tuning, reinforcement learning (GRPO/PPO), or LoRA training on cloud GPUs via Tinker's managed infrastructure instead of local compute.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Fine-Tuning, Tinker, LoRA, Reinforcement Learning, Supervised Learning, Cloud Training, Vision-Language Models]
+dependencies: [tinker, tinker-cookbook, chz, transformers>=4.40.0, datasets, numpy]
+---
+
+# Tinker API - Cloud LLM Fine-Tuning
+
+Expert guidance for fine-tuning large language models using Tinker's managed cloud training API. Tinker handles GPU allocation, model hosting, and distributed training — you write the training logic, Tinker runs it on cloud infrastructure.
+
+## When to Use This Skill
+
+**Use Tinker when you need to:**
+- Fine-tune models up to 235B parameters without managing GPU infrastructure
+- Run LoRA training on Qwen, Llama, DeepSeek, or GPT-OSS models
+- Train vision-language models (Qwen3-VL)
+- Implement custom RL loops (GRPO, PPO, importance sampling) on cloud GPUs
+- Iterate quickly with a training API that handles hardware provisioning
+
+**Do NOT use Tinker when:**
+- You need full fine-tuning (not LoRA) — Tinker only supports LoRA
+- You need to train custom architectures — Tinker supports specific model families
+- You want to use your own GPUs — use Axolotl, Unsloth, or LLaMA-Factory instead
+- You need offline/air-gapped training
+
+**Tinker vs Alternatives:**
+
+| Need | Use |
+|------|-----|
+| Managed cloud LoRA training | **Tinker** |
+| Local GPU fine-tuning | Axolotl, Unsloth, LLaMA-Factory |
+| Full parameter fine-tuning | DeepSpeed + Transformers |
+| RLHF with TRL locally | TRL + GRPO skill |
+| Quantized training | Unsloth, bitsandbytes |
+
+## Quick Reference
+
+| Topic | Reference |
+|-------|-----------|
+| Setup & Core Concepts | [Getting Started](references/getting-started.md) |
+| API Classes & Types | [API Reference](references/api-reference.md) |
+| Supervised Learning | [Supervised Learning](references/supervised-learning.md) |
+| RL Training | [Reinforcement Learning](references/reinforcement-learning.md) |
+| Loss Functions | [Loss Functions](references/loss-functions.md) |
+| Chat Templates | [Rendering](references/rendering.md) |
+| Models & LoRA | [Models & LoRA](references/models-and-lora.md) |
+| Example Scripts | [Recipes](references/recipes.md) |
+
+## Installation
+
+```bash
+pip install tinker tinker-cookbook
+export TINKER_API_KEY=your_api_key_here
+```
+
+---
+
+## Workflow 1: Supervised Fine-Tuning (Cookbook)
+
+Use this for standard SFT with JSONL or HuggingFace datasets.
+
+### Checklist
+- [ ] Prepare data in JSONL chat format (`{"messages": [...]}`)
+- [ ] Choose base model (see model table below)
+- [ ] Set hyperparameters (LR, batch size, epochs)
+- [ ] Run training via Cookbook
+- [ ] Monitor metrics (`train_mean_nll`, `test/nll`)
+- [ ] Save and deploy weights
+
+### Implementation
+
+```python
+import chz
+import asyncio
+from tinker_cookbook.supervised import train
+from tinker_cookbook.supervised.types import ChatDatasetBuilderCommonConfig
+from tinker_cookbook.supervised.data import FromConversationFileBuilder
+from tinker_cookbook.renderers import TrainOnWhat
+from tinker_cookbook.model_info import get_recommended_renderer_name
+
+model_name = "Qwen/Qwen3-30B-A3B"
+renderer_name = get_recommended_renderer_name(model_name)
+
+common_config = ChatDatasetBuilderCommonConfig(
+    model_name_for_tokenizer=model_name,
+    renderer_name=renderer_name,
+    max_length=2048,
+    batch_size=128,
+    train_on_what=TrainOnWhat.ALL_ASSISTANT_MESSAGES,
+)
+
+dataset_builder = FromConversationFileBuilder(
+    common_config=common_config,
+    file_path="training_data.jsonl",
+)
+
+blueprint = chz.Blueprint(train.Config).apply({
+    "log_path": "/tmp/sft-run",
+    "model_name": model_name,
+    "dataset_builder": dataset_builder,
+    "learning_rate": 2e-4,
+    "lr_schedule": "cosine",
+    "num_epochs": 3,
+    "lora_rank": 32,
+})
+
+config = blueprint.make()
+asyncio.run(train.main(config))
+```
+
+### Data Format
+
+JSONL with chat messages (one per line):
+```json
+{"messages": [{"role": "user", "content": "Translate to French: hello"}, {"role": "assistant", "content": "bonjour"}]}
+```
+
+### TrainOnWhat Options
+
+| Option | When to Use |
+|--------|-------------|
+| `ALL_ASSISTANT_MESSAGES` | Standard SFT, multi-turn conversations |
+| `LAST_ASSISTANT_MESSAGE` | Classification, chain-of-thought where only final answer matters |
+
+---
+
+## Workflow 2: Reinforcement Learning (GRPO-style)
+
+Use this for training with reward functions — math reasoning, format compliance, verifiable tasks.
+
+### Checklist
+- [ ] Define reward function(s) that return float scores
+- [ ] Choose group size (16 recommended)
+- [ ] Set up sampling → reward → training loop
+- [ ] Monitor `correct`, `format`, `reward/total`, KL divergence
+- [ ] Keep KL below 0.01 for stable training
+
+### Implementation (Cookbook)
+
+```python
+import asyncio
+import chz
+from tinker_cookbook.rl import train
+from tinker_cookbook.recipes.math_rl.math_env import Gsm8kDatasetBuilder
+from tinker_cookbook import model_info
+
+model_name = "meta-llama/Llama-3.1-8B"
+renderer_name = model_info.get_recommended_renderer_name(model_name)
+
+builder = Gsm8kDatasetBuilder(
+    batch_size=128,
+    group_size=16,
+    renderer_name=renderer_name,
+    model_name_for_tokenizer=model_name,
+)
+
+blueprint = chz.Blueprint(train.Config).apply({
+    "model_name": model_name,
+    "log_path": "/tmp/rl-run",
+    "dataset_builder": builder,
+    "learning_rate": 4e-5,
+    "max_tokens": 256,
+})
+
+config = blueprint.make()
+asyncio.run(train.main(config))
+```
+
+### Custom RL with Low-Level API
+
+For full control over sampling, reward computation, and advantage centering:
+
+```python
+import tinker
+from tinker import types
+from tinker.types.tensor_data import TensorData
+import torch
+
+service_client = tinker.ServiceClient()
+training_client = service_client.create_lora_training_client(
+    base_model="meta-llama/Llama-3.1-8B", rank=32
+)
+
+for batch_idx, batch_rows in enumerate(dataset):
+    path = training_client.save_weights_for_sampler(name=f"{batch_idx:06d}").result().path
+    sampling_client = service_client.create_sampling_client(model_path=path)
+
+    datums = []
+    for question, answer in batch_rows:
+        prompt = renderer.build_generation_prompt([{"role": "user", "content": question}])
+        prompt_tokens = prompt.to_ints()
+        result = sampling_client.sample(
+            prompt=prompt, num_samples=16,
+            sampling_params=types.SamplingParams(max_tokens=256, stop=renderer.get_stop_sequences()),
+        ).result()
+
+        rewards = [compute_reward(seq, answer) for seq in result.sequences]
+        mean_reward = sum(rewards) / len(rewards)
+        advantages = [r - mean_reward for r in rewards]
+        if all(a == 0 for a in advantages):
+            continue
+
+        for seq, advantage in zip(result.sequences, advantages):
+            tokens = prompt_tokens + seq.tokens
+            ob_len = len(prompt_tokens) - 1
+            datum = types.Datum(
+                model_input=types.ModelInput.from_ints(tokens=tokens[:-1]),
+                loss_fn_inputs={
+                    "target_tokens": TensorData.from_torch(torch.tensor(tokens[1:])),
+                    "logprobs": TensorData.from_torch(torch.tensor([0.0]*ob_len + list(seq.logprobs))),
+                    "advantages": TensorData.from_torch(torch.tensor([0.0]*ob_len + [advantage]*(len(tokens)-1-ob_len))),
+                },
+            )
+            datums.append(datum)
+
+    fwd_bwd = training_client.forward_backward(datums, loss_fn="importance_sampling")
+    optim = training_client.optim_step(types.AdamParams(learning_rate=4e-5))
+    fwd_bwd.result(); optim.result()
+```
+
+### Available RL Loss Functions
+
+| Loss | Use Case |
+|------|----------|
+| `importance_sampling` | Standard policy gradient with off-policy correction |
+| `ppo` | Clipped surrogate objective (PPO) |
+| `cispo` | Clipped importance sampling PO |
+| `dro` | Direct reward optimization with quadratic penalty |
+
+---
+
+## Available Models
+
+| Model | Type | Architecture | Train $/M tokens |
+|-------|------|-------------|------------------|
+| Qwen3-4B-Instruct-2507 | Instruction | Dense Compact | $0.22 |
+| Qwen3-8B | Hybrid | Dense Small | $0.40 |
+| Qwen3-30B-A3B | Hybrid | MoE Medium | $0.36 |
+| Qwen3-32B | Hybrid | Dense Medium | $1.47 |
+| Qwen3-VL-30B-A3B-Instruct | Vision | MoE Medium | $0.53 |
+| Llama-3.2-1B | Base | Dense Compact | $0.09 |
+| Llama-3.1-8B | Base | Dense Small | $0.40 |
+| Llama-3.1-70B | Base | Dense Large | $3.16 |
+| DeepSeek-V3.1 | Hybrid | MoE Large | $3.38 |
+| GPT-OSS-120B | Reasoning | MoE Medium | $0.52 |
+
+**Model Selection Tips:**
+- **Cost efficiency**: MoE models (Qwen3-30B-A3B at $0.36/M)
+- **Experimentation**: Start with 8B models
+- **Vision tasks**: Qwen3-VL-30B-A3B-Instruct
+- **Reasoning**: Hybrid or Reasoning models with chain-of-thought
+
+## LoRA Configuration
+
+Tinker exclusively uses LoRA. Default rank: 32.
+
+```python
+training_client = service_client.create_lora_training_client(
+    base_model="Qwen/Qwen3-30B-A3B",
+    rank=32,
+    train_attn=True,
+    train_mlp=True,
+    seed=42,
+)
+```
+
+**Critical**: LoRA needs 20-100x higher LR than full fine-tuning. Use `tinker_cookbook.hyperparam_utils.get_lr()` for recommended values.
+
+## Hyperparameter Guide
+
+| Parameter | SFT Default | RL Default | Notes |
+|-----------|-------------|------------|-------|
+| `learning_rate` | 2e-4 | 4e-5 | Use `get_lr(model_name)` for recommended |
+| `batch_size` | 128 | 128 | Smaller generally better for fine-tuning |
+| `lora_rank` | 32 | 32 | Higher rank = more capacity |
+| `group_size` | N/A | 16 | Rollouts per problem for RL |
+| `max_length` | 2048-32768 | N/A | Sequence length for SFT |
+| `max_tokens` | N/A | 256 | Max generation length for RL |
+| `num_epochs` | 1-3 | N/A | Training passes |
+| `lr_schedule` | cosine | N/A | LR decay schedule |
+
+## Cost Estimation & Usage Tracking
+
+### Pre-Training Cost Estimation
+
+**ALWAYS estimate cost before starting Tinker training.** Load the `tinker-training-cost` skill and use its pricing tables or the bundled calculator:
+
+```bash
+python scripts/calculate_cost.py training_data.jsonl --model Qwen3-8B --epochs 3 --json
+```
+
+Present the cost estimate to the user for approval before starting training.
+
+### Post-Training Usage Reporting
+
+After training completes, report usage for billing:
+
+```typescript
+// Called automatically by the CLI after Tinker training
+await SynSci.reportUsage({
+  service: "tinker",
+  event_type: "training",
+  model: "Qwen3-8B",
+  tokens_used: 5000000,
+  metadata: { run_id: "my-run", epochs: 3, dataset: "training_data.jsonl" },
+})
+```
+
+The CLI tracks all Tinker usage and reports it to the Synthetic Sciences dashboard for billing.
+
+## Common Issues
+
+| Problem | Solution |
+|---------|----------|
+| `TINKER_API_KEY` not set | `export TINKER_API_KEY=your_key` or check SynSci credential sync |
+| KL divergence > 0.01 | Reduce learning rate, check group size |
+| OOM on dataset loading | Use `StreamingSupervisedDatasetFromHFDataset` for large datasets |
+| Reward stuck at 0 | Debug reward function independently, check answer extraction |
+| All advantages = 0 | Increase group size, ensure reward variance across completions |
+| Wrong tokenizer | Use model-specific tokenizer (see Models & LoRA reference) |
+
+## Saving and Resuming
+
+```python
+sampling_path = training_client.save_weights_for_sampler(name="final").result().path
+sampling_client = service_client.create_sampling_client(model_path=sampling_path)
+
+resume_path = training_client.save_state(name="checkpoint").result().path
+training_client.load_state(resume_path)
+```
+
+## Common Imports
+
+```python
+import tinker
+from tinker import types
+from tinker.types import Datum, ModelInput, TensorData, AdamParams, SamplingParams
+
+import chz
+import asyncio
+from tinker_cookbook.supervised import train
+from tinker_cookbook.supervised.types import ChatDatasetBuilder, ChatDatasetBuilderCommonConfig
+from tinker_cookbook.supervised.data import (
+    SupervisedDatasetFromHFDataset,
+    StreamingSupervisedDatasetFromHFDataset,
+    FromConversationFileBuilder,
+    conversation_to_datum,
+)
+from tinker_cookbook.renderers import get_renderer, TrainOnWhat
+from tinker_cookbook.model_info import get_recommended_renderer_name
+from tinker_cookbook.tokenizer_utils import get_tokenizer
+```
+
+## External Resources
+
+- Documentation: https://tinker-docs.thinkingmachines.ai/
+- Cookbook Repo: https://github.com/thinking-machines-lab/tinker-cookbook
+- Console: https://tinker-console.thinkingmachines.ai

package/bin/skills/tinker/references/api-reference.md

@@ -0,0 +1,168 @@
+# API Reference
+
+## ServiceClient
+
+Main entry point for Tinker API.
+
+```python
+client = tinker.ServiceClient()
+```
+
+**Methods:**
+- `get_server_capabilities()` - Query supported models
+- `create_lora_training_client(base_model, rank=32, seed=None, train_mlp=True, train_attn=True, train_unembed=True)` - Create LoRA training client
+- `create_training_client_from_state(path)` - Resume from checkpoint (weights only)
+- `create_training_client_from_state_with_optimizer(path)` - Resume from checkpoint (weights + optimizer)
+- `create_sampling_client(model_path=None, base_model=None)` - Create inference client
+- `create_rest_client()` - Create REST API client
+
+## TrainingClient
+
+Client for training with forward/backward passes.
+
+**Methods:**
+- `forward(data, loss_fn, loss_fn_config=None)` - Forward pass without gradients
+- `forward_backward(data, loss_fn, loss_fn_config=None)` - Compute gradients
+- `forward_backward_custom(data, loss_fn)` - Custom loss function (1.5x FLOPs)
+- `optim_step(adam_params)` - Update parameters
+- `save_state(name)` - Save weights + optimizer state
+- `load_state(path)` - Load weights only
+- `load_state_with_optimizer(path)` - Load weights + optimizer
+- `save_weights_for_sampler(name)` - Save for inference
+- `save_weights_and_get_sampling_client(name)` - Save and get sampler
+- `get_info()` - Get model info
+- `get_tokenizer()` - Get tokenizer
+
+## SamplingClient
+
+Client for text generation.
+
+**Methods:**
+- `sample(prompt, num_samples, sampling_params, include_prompt_logprobs=False, topk_prompt_logprobs=None)` - Generate completions
+- `compute_logprobs(prompt)` - Get prompt logprobs
+
+## Core Types
+
+### Datum
+```python
+types.Datum(
+    model_input=ModelInput,
+    loss_fn_inputs={"target_tokens": ..., "weights": ...}
+)
+```
+
+### ModelInput
+```python
+types.ModelInput.from_ints(tokens=[1, 2, 3])  # From token list
+types.ModelInput(chunks=[EncodedTextChunk, ImageChunk, ...])  # Multi-modal
+model_input.to_ints()  # Convert to token list
+model_input.length()   # Total context length
+```
+
+### AdamParams
+```python
+types.AdamParams(
+    learning_rate=1e-4,
+    beta1=0.9,
+    beta2=0.95,
+    eps=1e-8,
+    weight_decay=0.0,
+    grad_clip_norm=0.0,  # 0 = no clipping
+)
+```
+
+### SamplingParams
+```python
+types.SamplingParams(
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.9,
+    top_k=-1,  # -1 = no limit
+    stop=["<|endoftext|>"],
+    seed=42,
+)
+```
+
+### TensorData
+```python
+types.TensorData.from_numpy(np.array([...]))
+types.TensorData.from_torch(torch.tensor([...]))
+tensor_data.to_numpy()
+tensor_data.to_torch()
+```
+
+### ImageChunk
+```python
+types.ImageChunk(
+    data=image_bytes,
+    format="png",  # or "jpeg"
+    expected_tokens=None,  # Optional advisory
+)
+```
+
+### EncodedTextChunk
+```python
+types.EncodedTextChunk(tokens=[1, 2, 3])
+```
+
+### LoraConfig
+```python
+types.LoraConfig(
+    rank=32,
+    seed=42,
+    train_unembed=False,
+    train_mlp=True,
+    train_attn=True,
+)
+```
+
+## Response Types
+
+### ForwardBackwardOutput
+```python
+result = fwdbwd_future.result()
+result.loss_fn_outputs  # List of dicts with "logprobs"
+result.metrics  # Training metrics
+```
+
+### SampleResponse
+```python
+result = sample_future.result()
+result.sequences  # List of SampledSequence
+result.prompt_logprobs  # If requested
+result.topk_prompt_logprobs  # If requested
+```
+
+### SampledSequence
+```python
+seq = result.sequences[0]
+seq.tokens  # Generated token IDs
+seq.logprobs  # Per-token logprobs
+seq.stop_reason  # Why generation stopped
+```
+
+### SaveWeightsResponse
+```python
+response = training_client.save_state(name).result()
+response.path  # "tinker://<model_id>/<name>"
+```
+
+## Checkpoint Types
+
+### Checkpoint
+```python
+checkpoint.checkpoint_id
+checkpoint.checkpoint_type  # "training" or "sampler"
+checkpoint.time
+checkpoint.tinker_path
+checkpoint.size_bytes
+checkpoint.public
+```
+
+### ParsedCheckpointTinkerPath
+```python
+parsed = ParsedCheckpointTinkerPath.from_tinker_path("tinker://...")
+parsed.training_run_id
+parsed.checkpoint_type
+parsed.checkpoint_id
+```

package/bin/skills/tinker/references/getting-started.md

@@ -0,0 +1,157 @@
+# Getting Started with Tinker
+
+## Creating Clients
+
+```python
+import tinker
+service_client = tinker.ServiceClient()
+
+# Check available models
+for item in service_client.get_server_capabilities().supported_models:
+    print("- " + item.model_name)
+
+# Create training client
+training_client = service_client.create_lora_training_client(
+    base_model="Qwen/Qwen3-VL-30B-A3B-Instruct",
+    rank=32,
+)
+
+# Get tokenizer
+tokenizer = training_client.get_tokenizer()
+```
+
+## Preparing Training Data
+
+```python
+import numpy as np
+from tinker import types
+
+def process_example(example: dict, tokenizer) -> types.Datum:
+    prompt = f"English: {example['input']}\nPig Latin:"
+    prompt_tokens = tokenizer.encode(prompt, add_special_tokens=True)
+
+    completion_tokens = tokenizer.encode(f" {example['output']}\n\n", add_special_tokens=False)
+    tokens = prompt_tokens + completion_tokens
+    weights = np.array(([0] * len(prompt_tokens)) + ([1] * len(completion_tokens)), dtype=np.float32)
+    target_tokens = np.array(tokens[1:], dtype=np.int64)
+
+    return types.Datum(
+        model_input=types.ModelInput.from_ints(tokens=tokens[:-1]),
+        loss_fn_inputs=dict(weights=weights[1:], target_tokens=target_tokens)
+    )
+```
+
+## Vision Inputs
+
+```python
+import requests
+
+image_data = requests.get("https://example.com/image.png").content
+model_input = tinker.ModelInput(chunks=[
+    types.EncodedTextChunk(tokens=tokenizer.encode("<|im_start|>user\n<|vision_start|>")),
+    types.ImageChunk(data=image_data, format="png"),
+    types.EncodedTextChunk(tokens=tokenizer.encode("<|vision_end|>What is this?<|im_end|>\n<|im_start|>assistant\n")),
+])
+```
+
+## Training Loop
+
+```python
+import numpy as np
+
+for _ in range(6):
+    fwdbwd_future = training_client.forward_backward(processed_examples, "cross_entropy")
+    optim_future = training_client.optim_step(types.AdamParams(learning_rate=1e-4))
+
+    fwdbwd_result = fwdbwd_future.result()
+    optim_result = optim_future.result()
+
+    # Compute loss
+    logprobs = np.concatenate([out['logprobs'].tolist() for out in fwdbwd_result.loss_fn_outputs])
+    weights = np.concatenate([ex.loss_fn_inputs['weights'].tolist() for ex in processed_examples])
+    print(f"Loss per token: {-np.dot(logprobs, weights) / weights.sum():.4f}")
+```
+
+## Sampling
+
+```python
+# Create sampling client
+sampling_client = training_client.save_weights_and_get_sampling_client(name='my-model')
+
+# Sample
+prompt = types.ModelInput.from_ints(tokens=tokenizer.encode("English: coffee break\nPig Latin:", add_special_tokens=True))
+params = types.SamplingParams(max_tokens=20, temperature=0.0, stop=["\n"])
+future = sampling_client.sample(prompt=prompt, sampling_params=params, num_samples=8)
+result = future.result()
+
+for i, seq in enumerate(result.sequences):
+    print(f"{i}: {repr(tokenizer.decode(seq.tokens))}")
+```
+
+## Computing Logprobs
+
+```python
+# Get prompt logprobs
+prompt = types.ModelInput.from_ints(tokens=tokenizer.encode("How many r's are in strawberry?", add_special_tokens=True))
+sample_response = sampling_client.sample(
+    prompt=prompt,
+    num_samples=1,
+    sampling_params=tinker.SamplingParams(max_tokens=1),
+    include_prompt_logprobs=True,
+).result()
+print(sample_response.prompt_logprobs)  # [None, -9.5, -1.6, ...]
+
+# Top-k logprobs
+sample_response = sampling_client.sample(
+    prompt=prompt,
+    num_samples=1,
+    sampling_params=tinker.SamplingParams(max_tokens=1),
+    include_prompt_logprobs=True,
+    topk_prompt_logprobs=5,
+).result()
+print(sample_response.topk_prompt_logprobs)  # [None, [(token_id, logprob), ...], ...]
+```
+
+## Async and Futures
+
+Every method has sync and async versions:
+
+| Sync | Async |
+|------|-------|
+| `create_lora_training_client()` | `create_lora_training_client_async()` |
+| `forward()` | `forward_async()` |
+| `sample()` | `sample_async()` |
+
+```python
+# Sync
+future = client.forward_backward(data, loss_fn)
+result = future.result()  # Blocks
+
+# Async (double await)
+future = await client.forward_backward_async(data, loss_fn)
+result = await future
+```
+
+### Overlap Requests for Performance
+
+```python
+# Submit both before waiting - runs in same clock cycle
+fwd_bwd_future = await client.forward_backward_async(batch, loss_fn)
+optim_future = await client.optim_step_async(adam_params)
+
+# Now retrieve results
+fwd_bwd_result = await fwd_bwd_future
+optim_result = await optim_future
+```
+
+## Saving and Loading
+
+```python
+# Save weights for sampling (fast, smaller)
+sampling_path = training_client.save_weights_for_sampler(name="0000").result().path
+sampling_client = service_client.create_sampling_client(model_path=sampling_path)
+
+# Save full state for resuming training
+resume_path = training_client.save_state(name="0010").result().path
+training_client.load_state(resume_path)
+```