@synsci/cli-darwin-x64 1.1.49

package/bin/skills/tinker/references/loss-functions.md

@@ -0,0 +1,163 @@
+# Loss Functions
+
+## Built-in Loss Functions
+
+Pass as string to `forward_backward(data, loss_fn)`.
+
+## Supervised Learning
+
+### cross_entropy
+
+Standard next-token prediction loss.
+
+$$\mathcal{L}(\theta) = -\mathbb{E}_x[\log p_\theta(x)]$$
+
+```python
+fwd_bwd = training_client.forward_backward(data, loss_fn="cross_entropy")
+```
+
+**Inputs:**
+- `target_tokens: array[(N,), int]` - Target token IDs
+- `weights: array[(N,), float]` - Loss weights (0 = ignore, 1 = train)
+
+**Outputs:**
+- `logprobs: array[(N,), float]` - Log probabilities
+- `loss:sum` (scalar) - Total weighted loss
+
+## Policy Gradient Methods
+
+### importance_sampling
+
+Policy gradient with importance weighting for off-policy correction:
+
+$$\mathcal{L}_{IS}(\theta) = \mathbb{E}_{x\sim q}\left[\frac{p_\theta(x)}{q(x)}A(x)\right]$$
+
+```python
+fwd_bwd = training_client.forward_backward(data, loss_fn="importance_sampling")
+```
+
+**Inputs:**
+- `target_tokens: array[(N,), int]` - Target tokens
+- `logprobs: array[(N,), float]` - Sampling logprobs (from q)
+- `advantages: array[(N,), float]` - Advantage values
+
+### ppo
+
+Proximal Policy Optimization with clipping:
+
+$$\mathcal{L}_{PPO}(\theta) = -\mathbb{E}_{x \sim q}\left[\min\left(\frac{p_\theta(x)}{q(x)} A(x), \text{clip}\left(\frac{p_\theta(x)}{q(x)}, 1-\epsilon, 1+\epsilon\right) A(x)\right)\right]$$
+
+```python
+fwd_bwd = training_client.forward_backward(
+    data,
+    loss_fn="ppo",
+    loss_fn_config={"clip_low_threshold": 0.9, "clip_high_threshold": 1.1}
+)
+```
+
+**Inputs:** Same as `importance_sampling`
+
+### cispo
+
+Clipped Importance Sampling Policy Optimization:
+
+$$\mathcal{L}_{CISPO}(\theta) = \mathbb{E}_{x \sim q}\left[\text{sg}\left(\text{clip}\left(\frac{p_\theta(x)}{q(x)}, 1-\epsilon, 1+\epsilon\right)\right) \log p_\theta(x) A(x)\right]$$
+
+```python
+fwd_bwd = training_client.forward_backward(
+    data,
+    loss_fn="cispo",
+    loss_fn_config={"clip_low_threshold": 0.8, "clip_high_threshold": 1.2}
+)
+```
+
+### dro
+
+Direct Reward Optimization with quadratic penalty:
+
+$$\mathcal{L}_{DRO}(\theta) = \mathbb{E}_{x \sim q}\left[\log p_\theta(x) A(x) - \frac{1}{2}\beta \left(\log \frac{p_\theta(x)}{q(x)}\right)^2\right]$$
+
+```python
+fwd_bwd = training_client.forward_backward(
+    data,
+    loss_fn="dro",
+    loss_fn_config={"beta": 0.05}
+)
+```
+
+## Custom Loss Functions
+
+For losses not covered above, use `forward_backward_custom`:
+
+```python
+def custom_loss(data: list[Datum], logprobs: list[torch.Tensor]) -> tuple[torch.Tensor, dict[str, float]]:
+    loss = (logprobs ** 2).sum()
+    return loss, {"custom_loss": loss.item()}
+
+loss, metrics = training_client.forward_backward_custom(data, custom_loss)
+```
+
+### Multi-Sequence Loss
+
+```python
+def variance_loss(data: list[Datum], logprobs: list[torch.Tensor]) -> tuple[torch.Tensor, dict[str, float]]:
+    flat_logprobs = torch.cat(logprobs)
+    variance = torch.var(flat_logprobs)
+    return variance, {"variance_loss": variance.item()}
+```
+
+### How forward_backward_custom Works
+
+1. Forward pass computes logprobs
+2. Custom function computes loss from logprobs
+3. `loss.backward()` computes grad_outputs
+4. Second forward_backward with linear surrogate loss
+
+**Note:** Uses 1.5x FLOPs and up to 3x wall time compared to built-in losses.
+
+## Implementation Details
+
+All losses:
+- Applied at token level with shape `(N,)` where N = sequence length
+- Use sum reduction (not mean)
+- Accept numpy.ndarray or torch.Tensor inputs
+
+### cross_entropy Implementation
+```python
+elementwise_loss = -target_logprobs * weights
+loss = elementwise_loss.sum()
+```
+
+### importance_sampling Implementation
+```python
+prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
+loss = -(prob_ratio * advantages).sum()
+```
+
+### ppo Implementation
+```python
+prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
+clipped_ratio = torch.clamp(prob_ratio, 1 - eps, 1 + eps)
+unclipped = prob_ratio * advantages
+clipped = clipped_ratio * advantages
+loss = -torch.min(unclipped, clipped).sum()
+```
+
+### cispo Implementation
+```python
+prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
+clipped_ratio = torch.clamp(prob_ratio, 1 - eps, 1 + eps)
+loss = -(clipped_ratio.detach() * target_logprobs * advantages).sum()
+```
+
+### dro Implementation
+```python
+quadratic_term = (target_logprobs - sampling_logprobs) ** 2
+loss = -(target_logprobs * advantages - 0.5 * beta * quadratic_term).sum()
+```
+
+## Notes
+
+- KL regularization: Include in reward rather than loss (see `incorporate_kl_penalty` in Cookbook)
+- Aggregation: Token-level losses are summed; for different schemes, modify advantages
+- Reference: [Schulman et al., 2017](https://arxiv.org/abs/1707.06347) for PPO

package/bin/skills/tinker/references/models-and-lora.md

@@ -0,0 +1,139 @@
+# Available Models & LoRA
+
+## Model Selection Guide
+
+- **Use MoE models** - More cost effective than dense
+- **Base models** - Only for research or full post-training
+- **Instruction models** - Fast inference, no chain-of-thought
+- **Hybrid/Reasoning models** - Long chain-of-thought for quality
+
+## Model Lineup
+
+| Model | Type | Architecture |
+|-------|------|--------------|
+| **Qwen/Qwen3-VL-235B-A22B-Instruct** | Vision | MoE Large |
+| **Qwen/Qwen3-VL-30B-A3B-Instruct** | Vision | MoE Medium |
+| Qwen/Qwen3-235B-A22B-Instruct-2507 | Instruction | MoE Large |
+| Qwen/Qwen3-30B-A3B-Instruct-2507 | Instruction | MoE Medium |
+| **Qwen/Qwen3-30B-A3B** | Hybrid | MoE Medium |
+| Qwen/Qwen3-30B-A3B-Base | Base | MoE Medium |
+| Qwen/Qwen3-32B | Hybrid | Dense Medium |
+| Qwen/Qwen3-8B | Hybrid | Dense Small |
+| Qwen/Qwen3-8B-Base | Base | Dense Small |
+| Qwen/Qwen3-4B-Instruct-2507 | Instruction | Dense Compact |
+| openai/gpt-oss-120b | Reasoning | MoE Medium |
+| openai/gpt-oss-20b | Reasoning | MoE Small |
+| deepseek-ai/DeepSeek-V3.1 | Hybrid | MoE Large |
+| deepseek-ai/DeepSeek-V3.1-Base | Base | MoE Large |
+| **meta-llama/Llama-3.1-8B** | Base | Dense Small |
+| meta-llama/Llama-3.1-8B-Instruct | Instruction | Dense Small |
+| meta-llama/Llama-3.3-70B-Instruct | Instruction | Dense Large |
+| meta-llama/Llama-3.1-70B | Base | Dense Large |
+| meta-llama/Llama-3.2-3B | Base | Dense Compact |
+| meta-llama/Llama-3.2-1B | Base | Dense Compact |
+| moonshotai/Kimi-K2-Thinking | Reasoning | MoE Large |
+
+**Sizes:** Compact (1-4B), Small (8B), Medium (30-32B), Large (70B+)
+
+**Types:**
+- **Base**: Pretrained, for post-training research
+- **Instruction**: Chat-tuned, fast inference
+- **Hybrid**: Thinking + non-thinking modes
+- **Reasoning**: Always uses chain-of-thought
+- **Vision**: VLMs with image processing
+
+## LoRA Primer
+
+LoRA (Low-Rank Adaptation) fine-tunes small parameter subset instead of all weights.
+
+### When LoRA Works Well
+
+- SL on small-medium instruction datasets: **Same as full fine-tuning**
+- RL: **Equivalent to full fine-tuning even with small ranks**
+- Large datasets: May underperform (increase rank)
+
+### LoRA Learning Rate
+
+**Critical:** LoRA needs 20-100x higher LR than full fine-tuning!
+
+```python
+from tinker_cookbook.hyperparam_utils import get_lora_lr_over_full_finetune_lr
+
+model_name = "meta-llama/Llama-3.1-8B"
+factor = get_lora_lr_over_full_finetune_lr(model_name)
+# Returns 10.0 for all models (empirically validated)
+```
+
+### Recommended Learning Rate
+
+```python
+from tinker_cookbook.hyperparam_utils import get_lr
+
+recommended_lr = get_lr("meta-llama/Llama-3.1-8B")
+```
+
+### LoRA Rank
+
+Default rank: 32
+
+```python
+from tinker_cookbook.hyperparam_utils import get_lora_param_count
+
+# Check parameter count
+param_count = get_lora_param_count("meta-llama/Llama-3.1-8B", lora_rank=32)
+```
+
+**Rule of thumb:** LoRA params ≥ completion tokens for good SL results.
+
+For RL: Small ranks work fine.
+
+**Optimal LR does NOT depend on rank** - same LR works across ranks.
+
+### LoRA Configuration
+
+```python
+training_client = service_client.create_lora_training_client(
+    base_model="meta-llama/Llama-3.1-8B",
+    rank=32,
+    train_attn=True,   # Attention layers (default)
+    train_mlp=True,    # MLP layers (default)
+    train_unembed=False,  # Output embedding (optional)
+    seed=42,  # For reproducibility
+)
+```
+
+**Best practice:** Train all layers (attention + MLP), not just attention.
+
+### Mathematical Definition
+
+Original weight: W (n×n)
+LoRA: W' = W + BA
+
+- B: n×r matrix
+- A: r×n matrix
+- r: rank (default 32)
+
+Think of LoRA as efficient random projection of parameter space.
+
+## Model Selection Tips
+
+1. **For cost efficiency:** Use MoE models (Qwen3-VL, Qwen3-30B-A3B)
+2. **For experimentation:** Start with 8B models
+3. **For vision tasks:** Qwen3-VL-30B-A3B-Instruct (cost-effective)
+4. **For reasoning:** Hybrid or Reasoning models with CoT
+5. **For latency:** Instruction models without CoT
+
+## Creating Training Client
+
+```python
+# Get available models
+service_client = tinker.ServiceClient()
+for model in service_client.get_server_capabilities().supported_models:
+    print(model.model_name)
+
+# Create training client
+training_client = service_client.create_lora_training_client(
+    base_model="Qwen/Qwen3-30B-A3B",
+    rank=32,
+)
+```

package/bin/skills/tinker/references/recipes.md

@@ -0,0 +1,280 @@
+# Example Recipes
+
+## sl_basic.py - Basic Supervised Learning
+
+```python
+import chz
+import sys
+import asyncio
+from tinker_cookbook import cli_utils, model_info
+from tinker_cookbook.recipes.chat_sl import chat_datasets
+from tinker_cookbook.renderers import TrainOnWhat
+from tinker_cookbook.supervised import train
+from tinker_cookbook.supervised.data import FromConversationFileBuilder
+from tinker_cookbook.supervised.types import ChatDatasetBuilderCommonConfig
+
+def build_config_blueprint() -> chz.Blueprint[train.Config]:
+    model_name = "meta-llama/Llama-3.1-8B"
+    renderer_name = model_info.get_recommended_renderer_name(model_name)
+    common_config = ChatDatasetBuilderCommonConfig(
+        model_name_for_tokenizer=model_name,
+        renderer_name=renderer_name,
+        max_length=32768,
+        batch_size=128,
+        train_on_what=TrainOnWhat.ALL_ASSISTANT_MESSAGES,
+    )
+    dataset = chat_datasets.NoRobotsBuilder(common_config=common_config)
+
+    # For custom dataset:
+    # dataset = FromConversationFileBuilder(
+    #     common_config=common_config,
+    #     file_path="/path/to/dataset.jsonl"
+    # )
+
+    return chz.Blueprint(train.Config).apply({
+        "log_path": "/tmp/tinker-examples/sl_basic",
+        "model_name": model_name,
+        "dataset_builder": dataset,
+        "learning_rate": 2e-4,
+        "lr_schedule": "linear",
+        "num_epochs": 1,
+        "eval_every": 8,
+    })
+
+def main(config: train.Config):
+    cli_utils.check_log_dir(config.log_path, behavior_if_exists="ask")
+    asyncio.run(train.main(config))
+
+if __name__ == "__main__":
+    blueprint = build_config_blueprint()
+    blueprint.make_from_argv(sys.argv[1:])
+    main(blueprint.make())
+```
+
+## sl_loop.py - Manual Training Loop
+
+```python
+import chz
+import datasets
+import tinker
+from tinker_cookbook import checkpoint_utils, model_info, renderers
+from tinker_cookbook.supervised.common import compute_mean_nll
+from tinker_cookbook.supervised.data import conversation_to_datum
+from tinker_cookbook.tokenizer_utils import get_tokenizer
+
+@chz.chz
+class Config:
+    log_path: str = "/tmp/tinker-examples/sl-loop"
+    model_name: str = "meta-llama/Llama-3.1-8B"
+    batch_size: int = 128
+    learning_rate: float = 1e-4
+    max_length: int = 32768
+    train_on_what: renderers.TrainOnWhat = renderers.TrainOnWhat.ALL_ASSISTANT_MESSAGES
+    lora_rank: int = 32
+
+def main(config: Config):
+    tokenizer = get_tokenizer(config.model_name)
+    renderer_name = model_info.get_recommended_renderer_name(config.model_name)
+    renderer = renderers.get_renderer(renderer_name, tokenizer)
+
+    dataset = datasets.load_dataset("HuggingFaceH4/no_robots")
+    train_dataset = dataset["train"].shuffle(seed=0)
+    n_batches = len(train_dataset) // config.batch_size
+
+    service_client = tinker.ServiceClient()
+    training_client = service_client.create_lora_training_client(
+        base_model=config.model_name, rank=config.lora_rank
+    )
+
+    for batch_idx in range(n_batches):
+        # Linear LR decay
+        lr_mult = max(0.0, 1.0 - batch_idx / n_batches)
+        adam_params = tinker.AdamParams(
+            learning_rate=config.learning_rate * lr_mult
+        )
+
+        # Get batch
+        start = batch_idx * config.batch_size
+        end = (batch_idx + 1) * config.batch_size
+        batch_rows = train_dataset.select(range(start, end))
+
+        batch = [
+            conversation_to_datum(row["messages"], renderer, config.max_length, config.train_on_what)
+            for row in batch_rows
+        ]
+
+        # Training step
+        fwd_bwd = training_client.forward_backward(batch, loss_fn="cross_entropy")
+        optim = training_client.optim_step(adam_params)
+        fwd_bwd_result = fwd_bwd.result()
+        optim.result()
+
+        # Compute metrics
+        train_nll = compute_mean_nll(
+            [x["logprobs"] for x in fwd_bwd_result.loss_fn_outputs],
+            [d.loss_fn_inputs["weights"] for d in batch]
+        )
+        print(f"Step {batch_idx}, NLL: {train_nll:.4f}")
+
+if __name__ == "__main__":
+    chz.nested_entrypoint(main)
+```
+
+## rl_basic.py - Basic RL
+
+```python
+import asyncio
+import chz
+import sys
+from tinker_cookbook import cli_utils, model_info
+from tinker_cookbook.recipes.math_rl.math_env import Gsm8kDatasetBuilder
+from tinker_cookbook.rl import train
+
+def build_config_blueprint() -> chz.Blueprint[train.Config]:
+    model_name = "meta-llama/Llama-3.1-8B"
+    renderer_name = model_info.get_recommended_renderer_name(model_name)
+    builder = Gsm8kDatasetBuilder(
+        batch_size=128,
+        group_size=16,
+        renderer_name=renderer_name,
+        model_name_for_tokenizer=model_name,
+    )
+    return chz.Blueprint(train.Config).apply({
+        "model_name": model_name,
+        "log_path": "/tmp/tinker-examples/rl_basic",
+        "dataset_builder": builder,
+        "learning_rate": 4e-5,
+        "max_tokens": 256,
+        "eval_every": 0,
+    })
+
+def main(config: train.Config):
+    cli_utils.check_log_dir(config.log_path, behavior_if_exists="ask")
+    asyncio.run(train.main(config))
+
+if __name__ == "__main__":
+    blueprint = build_config_blueprint()
+    blueprint.make_from_argv(sys.argv[1:])
+    main(blueprint.make())
+```
+
+## rl_loop.py - Manual RL Loop
+
+```python
+import chz
+import datasets
+import tinker
+from tinker import types
+from tinker.types.tensor_data import TensorData
+import torch
+from tinker_cookbook import model_info, renderers
+from tinker_cookbook.recipes.math_rl.math_grading import extract_boxed, grade_answer
+from tinker_cookbook.tokenizer_utils import get_tokenizer
+
+@chz.chz
+class Config:
+    model_name: str = "meta-llama/Llama-3.1-8B"
+    batch_size: int = 128
+    group_size: int = 16
+    learning_rate: float = 4e-5
+    max_tokens: int = 256
+
+def get_reward(response: str, answer: str) -> float:
+    try:
+        given = extract_boxed(response)
+        return 1.0 if grade_answer(given, answer) else 0.0
+    except ValueError:
+        return 0.0
+
+def main(config: Config):
+    tokenizer = get_tokenizer(config.model_name)
+    renderer = renderers.get_renderer(
+        model_info.get_recommended_renderer_name(config.model_name),
+        tokenizer
+    )
+
+    dataset = datasets.load_dataset("openai/gsm8k", "main")["train"]
+
+    service_client = tinker.ServiceClient()
+    training_client = service_client.create_lora_training_client(
+        base_model=config.model_name, rank=32
+    )
+
+    sampling_params = types.SamplingParams(
+        max_tokens=config.max_tokens,
+        stop=renderer.get_stop_sequences(),
+    )
+    adam_params = types.AdamParams(learning_rate=config.learning_rate)
+
+    for batch_idx in range(len(dataset) // config.batch_size):
+        # Save weights for sampling
+        path = training_client.save_weights_for_sampler(name=f"{batch_idx:06d}").result().path
+        sampling_client = service_client.create_sampling_client(model_path=path)
+
+        batch_start = batch_idx * config.batch_size
+        batch_rows = dataset.select(range(batch_start, batch_start + config.batch_size))
+
+        datums = []
+        for question, answer in zip(batch_rows["question"], batch_rows["answer"]):
+            convo = [{"role": "user", "content": question}]
+            prompt = renderer.build_generation_prompt(convo)
+            prompt_tokens = prompt.to_ints()
+
+            result = sampling_client.sample(
+                prompt=prompt,
+                num_samples=config.group_size,
+                sampling_params=sampling_params,
+            ).result()
+
+            rewards = [get_reward(renderers.get_text_content(renderer.parse_response(s.tokens)[0]), answer)
+                      for s in result.sequences]
+            mean_reward = sum(rewards) / len(rewards)
+            advantages = [r - mean_reward for r in rewards]
+
+            if all(a == 0 for a in advantages):
+                continue
+
+            for seq, advantage in zip(result.sequences, advantages):
+                tokens = prompt_tokens + seq.tokens
+                ob_len = len(prompt_tokens) - 1
+
+                datum = types.Datum(
+                    model_input=types.ModelInput.from_ints(tokens=tokens[:-1]),
+                    loss_fn_inputs={
+                        "target_tokens": TensorData.from_torch(torch.tensor(tokens[1:])),
+                        "logprobs": TensorData.from_torch(torch.tensor([0.0]*ob_len + list(seq.logprobs))),
+                        "advantages": TensorData.from_torch(torch.tensor([0.0]*ob_len + [advantage]*(len(tokens)-1-ob_len))),
+                    },
+                )
+                datums.append(datum)
+
+        fwd_bwd = training_client.forward_backward(datums, loss_fn="importance_sampling")
+        optim = training_client.optim_step(adam_params)
+        fwd_bwd.result()
+        optim.result()
+
+if __name__ == "__main__":
+    chz.nested_entrypoint(main)
+```
+
+## Running Recipes
+
+```bash
+# Basic SL
+python -m tinker_cookbook.recipes.sl_basic
+
+# Manual SL loop
+python -m tinker_cookbook.recipes.sl_loop
+
+# Basic RL
+python -m tinker_cookbook.recipes.rl_basic
+
+# Manual RL loop
+python -m tinker_cookbook.recipes.rl_loop
+```
+
+## CLI Overrides
+
+```bash
+python -m tinker_cookbook.recipes.sl_basic --learning_rate 1e-4 --batch_size 64
+```