@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/tinker/references/dpo-and-preference.md

@@ -1,174 +0,0 @@
-# DPO & Preference Learning
-
-## Direct Preference Optimization (DPO)
-
-DPO trains models to prefer chosen responses over rejected ones using a classification loss, without needing a separate reward model.
-
-### Quick Start
-
-```bash
-python -m tinker_cookbook.recipes.preference.train \
-    log_path=/tmp/dpo-experiment \
-    model_name=meta-llama/Llama-3.2-1B \
-    dataset=hhh \
-    renderer_name=role_colon \
-    learning_rate=1e-5 \
-    dpo_beta=0.1
-```
-
-### Key Parameters
-
-| Parameter | Description | Recommended |
-|-----------|-------------|-------------|
-| `model_name` | Base model (also used as reference policy) | Start with 1B-8B |
-| `dataset` | Preference dataset name | `hhh`, `helpsteer3`, `ultrafeedback` |
-| `renderer_name` | Chat format renderer | Match model family |
-| `learning_rate` | LR for optimization | 1e-5 to 1e-6 (lower than SFT) |
-| `dpo_beta` | Preference strength | Start with 0.1 |
-| `log_path` | Output directory | Required |
-
-### Available Datasets
-
-| Dataset | Source | Description |
-|---------|--------|-------------|
-| `hhh` | Anthropic | Helpful-Harmless-Honest pairwise comparisons |
-| `helpsteer3` | NVIDIA | HelpSteer3 preference dataset |
-| `ultrafeedback` | UltraFeedback | Binarized preferences |
-
-Custom datasets: implement `DPODatasetBuilder` from `tinker_cookbook.preference.preference_datasets`.
-
-### Training Metrics
-
-| Metric | Description | Watch For |
-|--------|-------------|-----------|
-| `dpo_loss` | DPO classification loss | Should decrease |
-| `accuracy` | Implicit reward model accuracy | Should increase |
-| `margin` | Chosen - rejected reward gap | Should increase |
-| `chosen_reward` | Average reward for chosen responses | Higher is better |
-| `rejected_reward` | Average reward for rejected responses | Lower is better |
-
-### Tips
-
-- **Beta parameter**: Start with `dpo_beta=0.1`, adjust based on dataset
-- **Learning rate**: Use lower LR than SFT (1e-5 to 1e-6)
-- **Base model**: Should already be in-distribution with the preference data. Either start with a light SFT phase or collect on-policy preferences. Sharp distribution mismatch creates strange behaviors
-- **Evaluation**: Use Inspect AI to evaluate after training (see [Evaluations](evaluations.md))
-
-### Evaluating DPO Models
-
-```bash
-MODEL_PATH=tinker://YOUR_MODEL_PATH_HERE
-python -m tinker_cookbook.eval.run_inspect_evals \
-    model_path=$MODEL_PATH \
-    model_name=meta-llama/Llama-3.2-1B \
-    tasks=inspect_evals/ifeval,inspect_evals/mmlu_0_shot
-```
-
-## RLHF Pipeline
-
-Full pipeline: SL → Preference Model → RL, implemented in `recipes/preference/rlhf/rlhf_pipeline.py`.
-
-```bash
-python -m recipes.preference.rlhf.rlhf_pipeline
-```
-
-### Step 1: Train Initial Policy (SL)
-
-Train on instruction-following data (e.g., no_robots from HuggingFace) to match InstructGPT methodology.
-
-### Step 2: Train Preference Model (SL)
-
-Train on pairwise comparison data (e.g., HHH dataset from Anthropic). Model sees completions A and B, outputs which is preferred.
-
-### Step 3: Train Policy via RL
-
-Use preference model as reward signal. For each prompt:
-1. Sample multiple completions from the policy
-2. Use preference model to grade all pairs
-3. Give reward based on win fraction (self-play)
-
-## Prompt Distillation
-
-Train a model to behave as if given a long prompt, without needing that prompt at inference time.
-
-### How It Works
-
-1. **Teacher generates data**: Long, detailed teacher prompt + queries → responses
-2. **Student trains on data**: Student model fine-tunes on (query, response) pairs without the teacher prompt
-
-### Example: Language Classification
-
-```bash
-# Step 1: Generate training data with teacher model
-python -m tinker_cookbook.recipes.prompt_distillation.create_data \
-    output_file=/tmp/tinker-datasets/prompt_distillation_lang.jsonl
-
-# Step 2: Train student model on distilled data
-python -m tinker_cookbook.recipes.prompt_distillation.train
-
-# Step 3: Test distilled model
-# Sample from trained model to verify performance
-```
-
-### When to Use
-
-- System prompt grows impractically long and model starts ignoring instructions
-- Need fast inference without long context overhead
-- Want to specialize a model for a narrow task distribution
-- Teacher and student can be the same model (self-distillation)
-
-### Advanced Configuration
-
-- **Teacher model selection**: Choose based on quality requirements
-- **Sampling strategies**: Adjust temperature and generation parameters
-- **Data volume**: Scale generated examples based on task complexity
-- **Training hyperparameters**: Follow standard SL hyperparameter guidance
-
-## LR Sweep Methodology
-
-For finding task-specific optimal LR:
-
-### Setup
-
-```python
-from tinker_cookbook.hyperparam_utils import get_lr
-default_lr = get_lr("meta-llama/Llama-3.1-8B")  # ~2.8e-4
-```
-
-### Sweep Range
-
-Sweep one order of magnitude above and below default:
-
-```bash
-# Launch in parallel (separate terminals)
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.003 log_path=/tmp/sweep/lr-0.003
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.001 log_path=/tmp/sweep/lr-0.001
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.0003 log_path=/tmp/sweep/lr-0.0003
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.0001 log_path=/tmp/sweep/lr-0.0001
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.00003 log_path=/tmp/sweep/lr-0.00003
-python -m tinker_cookbook.recipes.sl_loop learning_rate=0.00001 log_path=/tmp/sweep/lr-0.00001
-```
-
-### Collect and Visualize
-
-```python
-from glob import glob
-import pandas, json, os
-
-data = []
-for fname in sorted(glob("/tmp/sweep/*/metrics.jsonl")):
-    df = pandas.read_json(fname, lines=True)
-    if len(df) == 0 or df["progress"].iloc[-1] < 0.98:
-        continue
-    config = json.load(open(fname.replace("metrics.jsonl", "config.json")))
-    data.append({
-        "learning_rate": config["learning_rate"],
-        "final_loss": df["train_mean_nll"].iloc[-1].item()
-    })
-
-df = pandas.DataFrame(data)
-optimal_lr = df["learning_rate"][df["final_loss"].idxmin()]
-print(f"Optimal LR: {optimal_lr:.2e}")
-```
-
-Expected result: U-shaped curve with optimal LR near the `get_lr()` default.

package/bin/skills/tinker/references/evaluations.md

@@ -1,183 +0,0 @@
-# Evaluations
-
-## Inline Evals (During Training)
-
-Add evaluations that run periodically during training.
-
-### Supervised Fine-Tuning
-
-```python
-blueprint = chz.Blueprint(train.Config).apply({
-    "model_name": model_name,
-    "dataset_builder": dataset_builder,
-    "evaluator_builders": [my_evaluator],           # Runs every eval_every steps
-    "infrequent_evaluator_builders": [heavy_eval],   # Runs every infrequent_eval_every steps
-    "eval_every": 8,
-    "infrequent_eval_every": 50,
-})
-```
-
-### RL Training
-
-```python
-blueprint = chz.Blueprint(train.Config).apply({
-    "model_name": model_name,
-    "dataset_builder": builder,
-    "evaluator_builders": [sampling_eval],  # SamplingClientEvaluator instances
-    "eval_every": 5,
-})
-```
-
-## Offline Evals with Inspect AI
-
-Run standard evaluations on checkpoints using the Inspect AI library:
-
-```bash
-MODEL_PATH=tinker://YOUR_MODEL_PATH_HERE
-python -m tinker_cookbook.eval.run_inspect_evals \
-    model_path=$MODEL_PATH \
-    model_name=MODEL_NAME \
-    tasks=inspect_evals/ifeval,inspect_evals/mmlu_0_shot \
-    renderer_name=RENDERER_NAME
-```
-
-### Creating Custom Inspect AI Tasks
-
-```python
-import tinker
-from inspect_ai import Task, task
-from inspect_ai.dataset import MemoryDataset, Sample
-from inspect_ai.model import GenerateConfig as InspectAIGenerateConfig
-from inspect_ai.model import Model as InspectAIModel
-from inspect_ai.scorer import model_graded_qa
-from inspect_ai.solver import generate
-from tinker_cookbook.eval.inspect_utils import InspectAPIFromTinkerSampling
-
-QA_DATASET = MemoryDataset(
-    name="qa_dataset",
-    samples=[
-        Sample(input="What is the capital of France?", target="Paris"),
-        Sample(input="What is the capital of Italy?", target="Rome"),
-    ],
-)
-
-service_client = tinker.ServiceClient()
-sampling_client = service_client.create_sampling_client(
-    base_model="meta-llama/Llama-3.1-8B-Instruct"
-)
-
-api = InspectAPIFromTinkerSampling(
-    renderer_name="llama3",
-    model_name="meta-llama/Llama-3.1-8B-Instruct",
-    sampling_client=sampling_client,
-    verbose=False,
-)
-
-GRADER_MODEL = InspectAIModel(api=api, config=InspectAIGenerateConfig())
-
-@task
-def example_lm_as_judge() -> Task:
-    return Task(
-        name="llm_as_judge",
-        dataset=QA_DATASET,
-        solver=generate(),
-        scorer=model_graded_qa(
-            instructions="Grade strictly. Respond 'GRADE: C' if correct or 'GRADE: I' otherwise.",
-            partial_credit=False,
-            model=GRADER_MODEL,
-        ),
-    )
-```
-
-Inspect also supports any OpenAI-compatible API (e.g., openrouter) as the grader model.
-
-## Custom SamplingClientEvaluator
-
-Lower-level abstraction with fine-grained control:
-
-```python
-from typing import Any, Callable
-import tinker
-from tinker import types
-from tinker_cookbook import renderers
-from tinker_cookbook.evaluators import SamplingClientEvaluator
-from tinker_cookbook.tokenizer_utils import get_tokenizer
-
-class CustomEvaluator(SamplingClientEvaluator):
-    def __init__(
-        self,
-        dataset: Any,
-        grader_fn: Callable[[str, str], bool],
-        model_name: str,
-        renderer_name: str,
-    ):
-        self.dataset = dataset
-        self.grader_fn = grader_fn
-        tokenizer = get_tokenizer(model_name)
-        self.renderer = renderers.get_renderer(name=renderer_name, tokenizer=tokenizer)
-
-    async def __call__(self, sampling_client: tinker.SamplingClient) -> dict[str, float]:
-        sampling_params = types.SamplingParams(
-            max_tokens=100,
-            temperature=0.7,
-            top_p=1.0,
-            stop=self.renderer.get_stop_sequences(),
-        )
-
-        num_correct = 0
-        for datum in self.dataset:
-            model_input = self.renderer.build_generation_prompt(
-                [renderers.Message(role="user", content=datum["input"])]
-            )
-            r = await sampling_client.sample_async(
-                prompt=model_input, num_samples=1, sampling_params=sampling_params
-            )
-            tokens = r.sequences[0].tokens
-            response = self.renderer.parse_response(tokens)[0]
-            if self.grader_fn(response["content"], datum["output"]):
-                num_correct += 1
-
-        return {"accuracy": num_correct / len(self.dataset)}
-```
-
-### Usage
-
-```python
-import asyncio
-
-QA_DATASET = [
-    {"input": "What is the capital of France?", "output": "Paris"},
-    {"input": "What is the capital of Germany?", "output": "Berlin"},
-]
-
-def grader_fn(response: str, target: str) -> bool:
-    return target.lower() in response.lower()
-
-evaluator = CustomEvaluator(
-    dataset=QA_DATASET,
-    grader_fn=grader_fn,
-    renderer_name="llama3",
-    model_name="meta-llama/Llama-3.1-8B-Instruct",
-)
-
-service_client = tinker.ServiceClient()
-sampling_client = service_client.create_sampling_client(
-    base_model="meta-llama/Llama-3.1-8B-Instruct"
-)
-
-async def main():
-    result = await evaluator(sampling_client)
-    print(result)  # {"accuracy": 1.0}
-
-asyncio.run(main())
-```
-
-## Evaluation Strategy
-
-| Stage | Method | When |
-|-------|--------|------|
-| During SFT | `evaluator_builders` | Every N training steps |
-| During RL | `evaluator_builders` (SamplingClientEvaluator) | Every N iterations |
-| After training | `run_inspect_evals` CLI | On final checkpoint |
-| Custom tasks | Custom `SamplingClientEvaluator` | Any time with sampling client |
-| LLM-as-judge | Inspect AI with `model_graded_qa` | When automated grading needed |

package/bin/skills/tinker/references/getting-started.md

@@ -1,157 +0,0 @@
-# Getting Started with Tinker
-
-## Creating Clients
-
-```python
-import tinker
-service_client = tinker.ServiceClient()
-
-# Check available models
-for item in service_client.get_server_capabilities().supported_models:
-    print("- " + item.model_name)
-
-# Create training client
-training_client = service_client.create_lora_training_client(
-    base_model="Qwen/Qwen3-VL-30B-A3B-Instruct",
-    rank=32,
-)
-
-# Get tokenizer
-tokenizer = training_client.get_tokenizer()
-```
-
-## Preparing Training Data
-
-```python
-import numpy as np
-from tinker import types
-
-def process_example(example: dict, tokenizer) -> types.Datum:
-    prompt = f"English: {example['input']}\nPig Latin:"
-    prompt_tokens = tokenizer.encode(prompt, add_special_tokens=True)
-
-    completion_tokens = tokenizer.encode(f" {example['output']}\n\n", add_special_tokens=False)
-    tokens = prompt_tokens + completion_tokens
-    weights = np.array(([0] * len(prompt_tokens)) + ([1] * len(completion_tokens)), dtype=np.float32)
-    target_tokens = np.array(tokens[1:], dtype=np.int64)
-
-    return types.Datum(
-        model_input=types.ModelInput.from_ints(tokens=tokens[:-1]),
-        loss_fn_inputs=dict(weights=weights[1:], target_tokens=target_tokens)
-    )
-```
-
-## Vision Inputs
-
-```python
-import requests
-
-image_data = requests.get("https://example.com/image.png").content
-model_input = tinker.ModelInput(chunks=[
-    types.EncodedTextChunk(tokens=tokenizer.encode("<|im_start|>user\n<|vision_start|>")),
-    types.ImageChunk(data=image_data, format="png"),
-    types.EncodedTextChunk(tokens=tokenizer.encode("<|vision_end|>What is this?<|im_end|>\n<|im_start|>assistant\n")),
-])
-```
-
-## Training Loop
-
-```python
-import numpy as np
-
-for _ in range(6):
-    fwdbwd_future = training_client.forward_backward(processed_examples, "cross_entropy")
-    optim_future = training_client.optim_step(types.AdamParams(learning_rate=2e-4))  # Use get_lr() for production
-
-    fwdbwd_result = fwdbwd_future.result()
-    optim_result = optim_future.result()
-
-    # Compute loss
-    logprobs = np.concatenate([out['logprobs'].tolist() for out in fwdbwd_result.loss_fn_outputs])
-    weights = np.concatenate([ex.loss_fn_inputs['weights'].tolist() for ex in processed_examples])
-    print(f"Loss per token: {-np.dot(logprobs, weights) / weights.sum():.4f}")
-```
-
-## Sampling
-
-```python
-# Create sampling client
-sampling_client = training_client.save_weights_and_get_sampling_client(name='my-model')
-
-# Sample
-prompt = types.ModelInput.from_ints(tokens=tokenizer.encode("English: coffee break\nPig Latin:", add_special_tokens=True))
-params = types.SamplingParams(max_tokens=20, temperature=0.0, stop=["\n"])
-future = sampling_client.sample(prompt=prompt, sampling_params=params, num_samples=8)
-result = future.result()
-
-for i, seq in enumerate(result.sequences):
-    print(f"{i}: {repr(tokenizer.decode(seq.tokens))}")
-```
-
-## Computing Logprobs
-
-```python
-# Get prompt logprobs
-prompt = types.ModelInput.from_ints(tokens=tokenizer.encode("How many r's are in strawberry?", add_special_tokens=True))
-sample_response = sampling_client.sample(
-    prompt=prompt,
-    num_samples=1,
-    sampling_params=tinker.SamplingParams(max_tokens=1),
-    include_prompt_logprobs=True,
-).result()
-print(sample_response.prompt_logprobs)  # [None, -9.5, -1.6, ...]
-
-# Top-k logprobs
-sample_response = sampling_client.sample(
-    prompt=prompt,
-    num_samples=1,
-    sampling_params=tinker.SamplingParams(max_tokens=1),
-    include_prompt_logprobs=True,
-    topk_prompt_logprobs=5,
-).result()
-print(sample_response.topk_prompt_logprobs)  # [None, [(token_id, logprob), ...], ...]
-```
-
-## Async and Futures
-
-Every method has sync and async versions:
-
-| Sync | Async |
-|------|-------|
-| `create_lora_training_client()` | `create_lora_training_client_async()` |
-| `forward()` | `forward_async()` |
-| `sample()` | `sample_async()` |
-
-```python
-# Sync
-future = client.forward_backward(data, loss_fn)
-result = future.result()  # Blocks
-
-# Async (double await)
-future = await client.forward_backward_async(data, loss_fn)
-result = await future
-```
-
-### Overlap Requests for Performance
-
-```python
-# Submit both before waiting - runs in same clock cycle
-fwd_bwd_future = await client.forward_backward_async(batch, loss_fn)
-optim_future = await client.optim_step_async(adam_params)
-
-# Now retrieve results
-fwd_bwd_result = await fwd_bwd_future
-optim_result = await optim_future
-```
-
-## Saving and Loading
-
-```python
-# Save weights for sampling (fast, smaller)
-sampling_path = training_client.save_weights_for_sampler(name="0000").result().path
-sampling_client = service_client.create_sampling_client(model_path=sampling_path)
-
-# Save full state for resuming training
-resume_path = training_client.save_state(name="0010").result().path
-training_client.load_state(resume_path)
-```

package/bin/skills/tinker/references/loss-functions.md

@@ -1,163 +0,0 @@
-# Loss Functions
-
-## Built-in Loss Functions
-
-Pass as string to `forward_backward(data, loss_fn)`.
-
-## Supervised Learning
-
-### cross_entropy
-
-Standard next-token prediction loss.
-
-$$\mathcal{L}(\theta) = -\mathbb{E}_x[\log p_\theta(x)]$$
-
-```python
-fwd_bwd = training_client.forward_backward(data, loss_fn="cross_entropy")
-```
-
-**Inputs:**
-- `target_tokens: array[(N,), int]` - Target token IDs
-- `weights: array[(N,), float]` - Loss weights (0 = ignore, 1 = train)
-
-**Outputs:**
-- `logprobs: array[(N,), float]` - Log probabilities
-- `loss:sum` (scalar) - Total weighted loss
-
-## Policy Gradient Methods
-
-### importance_sampling
-
-Policy gradient with importance weighting for off-policy correction:
-
-$$\mathcal{L}_{IS}(\theta) = \mathbb{E}_{x\sim q}\left[\frac{p_\theta(x)}{q(x)}A(x)\right]$$
-
-```python
-fwd_bwd = training_client.forward_backward(data, loss_fn="importance_sampling")
-```
-
-**Inputs:**
-- `target_tokens: array[(N,), int]` - Target tokens
-- `logprobs: array[(N,), float]` - Sampling logprobs (from q)
-- `advantages: array[(N,), float]` - Advantage values
-
-### ppo
-
-Proximal Policy Optimization with clipping:
-
-$$\mathcal{L}_{PPO}(\theta) = -\mathbb{E}_{x \sim q}\left[\min\left(\frac{p_\theta(x)}{q(x)} A(x), \text{clip}\left(\frac{p_\theta(x)}{q(x)}, 1-\epsilon, 1+\epsilon\right) A(x)\right)\right]$$
-
-```python
-fwd_bwd = training_client.forward_backward(
-    data,
-    loss_fn="ppo",
-    loss_fn_config={"clip_low_threshold": 0.9, "clip_high_threshold": 1.1}
-)
-```
-
-**Inputs:** Same as `importance_sampling`
-
-### cispo
-
-Clipped Importance Sampling Policy Optimization:
-
-$$\mathcal{L}_{CISPO}(\theta) = \mathbb{E}_{x \sim q}\left[\text{sg}\left(\text{clip}\left(\frac{p_\theta(x)}{q(x)}, 1-\epsilon, 1+\epsilon\right)\right) \log p_\theta(x) A(x)\right]$$
-
-```python
-fwd_bwd = training_client.forward_backward(
-    data,
-    loss_fn="cispo",
-    loss_fn_config={"clip_low_threshold": 0.8, "clip_high_threshold": 1.2}
-)
-```
-
-### dro
-
-Direct Reward Optimization with quadratic penalty:
-
-$$\mathcal{L}_{DRO}(\theta) = \mathbb{E}_{x \sim q}\left[\log p_\theta(x) A(x) - \frac{1}{2}\beta \left(\log \frac{p_\theta(x)}{q(x)}\right)^2\right]$$
-
-```python
-fwd_bwd = training_client.forward_backward(
-    data,
-    loss_fn="dro",
-    loss_fn_config={"beta": 0.05}
-)
-```
-
-## Custom Loss Functions
-
-For losses not covered above, use `forward_backward_custom`:
-
-```python
-def custom_loss(data: list[Datum], logprobs: list[torch.Tensor]) -> tuple[torch.Tensor, dict[str, float]]:
-    loss = (logprobs ** 2).sum()
-    return loss, {"custom_loss": loss.item()}
-
-loss, metrics = training_client.forward_backward_custom(data, custom_loss)
-```
-
-### Multi-Sequence Loss
-
-```python
-def variance_loss(data: list[Datum], logprobs: list[torch.Tensor]) -> tuple[torch.Tensor, dict[str, float]]:
-    flat_logprobs = torch.cat(logprobs)
-    variance = torch.var(flat_logprobs)
-    return variance, {"variance_loss": variance.item()}
-```
-
-### How forward_backward_custom Works
-
-1. Forward pass computes logprobs
-2. Custom function computes loss from logprobs
-3. `loss.backward()` computes grad_outputs
-4. Second forward_backward with linear surrogate loss
-
-**Note:** Uses 1.5x FLOPs and up to 3x wall time compared to built-in losses.
-
-## Implementation Details
-
-All losses:
-- Applied at token level with shape `(N,)` where N = sequence length
-- Use sum reduction (not mean)
-- Accept numpy.ndarray or torch.Tensor inputs
-
-### cross_entropy Implementation
-```python
-elementwise_loss = -target_logprobs * weights
-loss = elementwise_loss.sum()
-```
-
-### importance_sampling Implementation
-```python
-prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
-loss = -(prob_ratio * advantages).sum()
-```
-
-### ppo Implementation
-```python
-prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
-clipped_ratio = torch.clamp(prob_ratio, 1 - eps, 1 + eps)
-unclipped = prob_ratio * advantages
-clipped = clipped_ratio * advantages
-loss = -torch.min(unclipped, clipped).sum()
-```
-
-### cispo Implementation
-```python
-prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
-clipped_ratio = torch.clamp(prob_ratio, 1 - eps, 1 + eps)
-loss = -(clipped_ratio.detach() * target_logprobs * advantages).sum()
-```
-
-### dro Implementation
-```python
-quadratic_term = (target_logprobs - sampling_logprobs) ** 2
-loss = -(target_logprobs * advantages - 0.5 * beta * quadratic_term).sum()
-```
-
-## Notes
-
-- KL regularization: Include in reward rather than loss (see `incorporate_kl_penalty` in Cookbook)
-- Aggregation: Token-level losses are summed; for different schemes, modify advantages
-- Reference: [Schulman et al., 2017](https://arxiv.org/abs/1707.06347) for PPO