runbios-mcp 0.5.0 → 0.6.0-dev.368

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/server.js CHANGED
@@ -844,8 +844,6 @@ launchGates = resolveLaunchGates(), allowedTools) {
844
844
  epochs: z.number().positive().max(1000).optional().describe("Number of training epochs. Governs the full run when max_steps is omitted (the default)."),
845
845
  max_steps: z.number().int().min(1).max(10_000_000).optional().describe("Optional hard step cap that overrides epochs. Omit for no cap and all configured epochs; setting less than one epoch intentionally trains only part of the data."),
846
846
  learning_rate: z.number().positive().max(1).optional().describe("Learning rate."),
847
- batch_size: z.number().int().min(1).max(65536).optional().describe("Per-device train batch size."),
848
- gradient_accumulation_steps: z.number().int().min(1).max(65536).optional(),
849
847
  max_seq_length: trainingSeqLengthSchema(),
850
848
  lora_rank: z.number().int().min(1).max(4096).optional(),
851
849
  lora_alpha: z.number().positive().max(1_000_000).optional(),
@@ -857,6 +855,12 @@ launchGates = resolveLaunchGates(), allowedTools) {
857
855
  .describe("Learning-rate scheduler."),
858
856
  job_name: z.string().max(255).optional(),
859
857
  eval_split_ratio: z.number().min(0).max(0.99).optional(),
858
+ validation_rows: z.number().int().min(0).max(50_000_000).optional()
859
+ .describe("Validation rows held out and never trained on (0 turns validation off). Allowed: 100 (or 5% when that is fewer) up to 40% of the training rows; unset, 5% capped at 1,000. Preflight returns the exact range. Do not send with eval_split_ratio."),
860
+ precision: z.enum(["bf16", "fp8"]).optional()
861
+ .describe("Training precision. Offer only what get_managed_training_options lists for the model and adapter; bf16 is the default."),
862
+ optim: z.enum(["adamw_torch_fused", "adamw_8bit"]).optional()
863
+ .describe("Optimizer state: adamw_torch_fused (16-bit, default, most accurate) or adamw_8bit (less memory). Offer only what get_managed_training_options lists."),
860
864
  eval_steps: z.number().int().positive().optional().describe("Evaluate every N optimizer steps when a validation split exists."),
861
865
  save_steps: z.number().int().positive().optional().describe("Checkpoint interval in optimizer steps."),
862
866
  early_stopping_patience: z.number().int().min(0).max(20).optional()
@@ -918,7 +922,7 @@ on.
918
922
  8. loop_list_models - The models a pipeline can train, and the platform's fixed sample minimums (floors). The only list a pipeline's model may come from; unavailable says, in one plain sentence per model, why one is not offered.
919
923
  9. loop_create_pipeline - One use case: a name, a model (optionally a second model, a different one), its tags, when to train (optionally a schedule; the sample minimums are fixed by the platform) and how a new version goes live. It switches on the source named by its pipeline ID, so an app can send samples at once. SPEND CONSENT: saving it authorises trainings on the workspace's wallet whenever its data has grown enough; the platform chooses the GPUs, and what is charged is the GPU time its trainings use and, where the platform scores samples, the judge's scoring of unreviewed samples, shown on the training it was for. Say so and wait for a yes.
920
924
  10. loop_list_pipelines / loop_get_pipeline / loop_list_pipeline_attempts / loop_update_pipeline / loop_delete_pipeline - Each pipeline's versions (only trainings that went live: v1, v2 ...: version version numbers count up; nothing untested goes live any more, and each version's tested says whether it was), its trainings and how each ended (the latest on the pipeline, every one a page at a time with loop_list_pipeline_attempts, each with an explanation in plain words), the live version and whether its deployment answers (live_deployment), status and next_reason, and its settings. Deleting a pipeline stops its live version's deployment (unless another pipeline or model name answers from it too) and deletes its history.
921
- 11. loop_run_pipeline / loop_import_into_pipeline - Train now (skips the schedule, never the minimum), and bring a file in as this pipeline's samples.
925
+ 11. loop_run_pipeline / loop_import_into_pipeline - Train now (skips the schedule, never the minimum), and bring a file in as this pipeline's samples. loop_list_live_deployments / loop_use_live_deployment - when the live version's deployment was deleted, find a deployment of its weights and answer the app from it; nothing is trained again.
922
926
  12. loop_create_benchmark / loop_list_benchmarks / loop_get_benchmark / loop_list_benchmark_items / loop_get_benchmark_history / loop_retire_benchmark / loop_get_benchmark_run / loop_list_benchmark_run_items / loop_benchmark_questions - Benchmarks: fixed sets of the user's questions with right answers that every training is tested on, so "is it getting better" has a yardstick that does not move, and each benchmark run's report: its scores, how many questions each model got right (score 0.70 or more), and every question behind them. loop_benchmark_questions shows each question with how a pipeline's recent trainings did on it.
923
927
  13. loop_list_pipeline_benchmarks / loop_attach_pipeline_benchmark / loop_move_pipeline_benchmark / loop_remove_pipeline_benchmark / loop_compare_versions / loop_compare_attempts - A pipeline's benchmarks in priority order (1 is the primary), its versions side by side (v1 against v5), and any two trainings side by side, live or not, question by question on the automatic-test questions both were scored on.
924
928
  14. loop_list_training_runs / loop_get_training_run / loop_get_training_curve - Every training of every pipeline (result filters by what became of it; each item's 'attempt' is the training as its pipeline's own list says it): its timeline, why it went live or did not (its going-live record, and verdict_checks: every bar its test needed, what it got and whether it was met), what each part cost, and available_actions — what this reader may actually do next, rather than what the state looks like it allows. loop_get_training_curve shows how one went while it trained, step by step, and the step it kept.
@@ -1172,6 +1176,7 @@ get_inference_status instead of asserting them.
1172
1176
  5. get_model_config - Get recommended settings for your chosen model
1173
1177
  6. list_supported_architectures - Check which architectures can be trained or served
1174
1178
  7. get_training_capabilities - Read the authoritative training contract; call this before building automated payloads
1179
+ 7a. get_managed_training_options - With managed_training on: the context lengths, precisions and optimizer states a model can be trained with; offer only these
1175
1180
  8. get_gpu_pricing - See GPU options and pricing
1176
1181
  9. get_recommended_gpu - Read default-config sizing; use full preflight for the actual run
1177
1182
  9a. recommend_training_config - Ask the trainer's sizing model for batch, accumulation, LR, predicted peak memory, minimum GPU count and wall clock on the GPU you picked; check your own batch size against it
@@ -1232,25 +1237,18 @@ Option B: Import from HuggingFace — call import_huggingface_dataset with a rep
1232
1237
 
1233
1238
  Dataset upload accepts one file type: JSONL. SFT rows may use messages, conversations, instruction/output, prompt/completion, query/response, input/output, question/answer, or problem/solution. CPT rows require a non-empty text field. Preference, reward, KTO, GRPO, PPO, and GKD datasets are refused while those training methods are unavailable.
1234
1239
 
1235
- ### Step 4: Choose method, adapter, and the complete training configuration
1236
- Call get_model_config and get_training_capabilities. Choose SFT or CPT, adapter, sequence length, per-device batch, gradient accumulation, optimizer, precision, epochs/optional max_steps, validation and checkpoint settings BEFORE asking for a GPU. Empty max_steps means no cap: all configured epochs train. Early stopping defaults on when validation exists.
1240
+ ### Managed training (check first)
1241
+ Call get_training_capabilities. If its features list has managed_training enabled, the platform chooses the hardware: do NOT choose or send a GPU, GPU count, per-device batch, gradient accumulation, recompute level, checkpoint count or price cap -- they are ignored (preflight warns FIELD_IGNORED). Call get_managed_training_options for the model, method and adapter, and offer only what it returns: the context length (max_length, one of 2048-131072), precision (bf16 or fp8) and optimizer state (optim: adamw_torch_fused = 16-bit, adamw_8bit = 8-bit). Send only: model, method (sft/cpt), adapter (lora/qlora/full), dataset(s), max_length, precision, optim, learning_rate, lr_scheduler_type, warmup_ratio, weight_decay, epochs or max_steps, lora_rank (8-64), lora_alpha, and optionally validation_rows (0 turns validation off) and early_stopping_patience. On preflight_training_job and create_training_job these are the inputs max_seq_length, precision, optim, scheduler, validation_rows. Validation rows must be between 100 (or 5% of the rows when that is under 100) and 40% of the training rows; unset, the job holds out 5% capped at 1,000 rows, and preflight returns the exact rows and range in validation. Unset, the schedule is cosine; it keeps its best (lowest validation loss) and latest checkpoints, is charged per trained token only, and waits unbilled in the queue, first come first served, when no machine is free (the customer is emailed when it starts and may cancel it while queued). Training logs and machine metrics (memory, MFU, throughput) are not available to the customer: report progress from get_training_status and training metrics from get_training_metrics. Samples longer than max_length lose their oldest turns, never the answer; image samples that do not fit are skipped. Preflight returns the placement validity and, when token_billing is enabled, an estimate (estimated tokens and cost at the model's per-token price; the charge is the exact tokens trained). Customers never see hardware; do not describe GPUs to them.
1237
1242
 
1238
- ### Step 5: Preflight the complete request and choose a GPU LAST
1239
- Call preflight_training_job with the model, dataset(s), method, adapter and complete config. Its GPU choices are sized from ALL of those inputs and joined to live inventory. Choose only a returned configuration; never guess a GPU count or use a choice below min_gpus.
1243
+ ### Step 4: Preflight
1244
+ Call preflight_training_job with the model, dataset(s), method, adapter and training settings. It returns valid, the validation rows and their allowed range, warnings and the per-token cost estimate. Show the estimate to the user before creating.
1240
1245
 
1241
- ### Step 6: Create the training job
1242
- Call create_training_job with:
1243
- - model: the model name from step 2
1244
- - dataset_id: the ID from step 3
1245
- - method: "sft" for instruction tuning (most common)
1246
- - adapter: "lora" (fastest, recommended for first jobs)
1247
- - Leave other params as defaults unless you have specific needs
1246
+ ### Step 5: Create the training job
1247
+ Call create_training_job with the same request. The job is placed on tested machines; when none is free it is queued (first come first served, unbilled) and starts automatically. The user is emailed when it starts and can cancel at any time with stop_training_job.
1248
1248
 
1249
- ### Step 7: Monitor progress
1250
- create_training_job books a GPU before returning (~40s). A 'booked' status means the GPU is secured (booked==secured) and training is starting; 'securing' means it is still booking — poll get_training_status until 'booked'/'running'. A booking-time capacity miss returns CAPACITY_UNAVAILABLE and no job is created.
1251
- Call get_training_status periodically. Training typically takes 1-4 hours for small models.
1252
- If a started job loses its pod it rests at 'interrupted' (billing already stopped, not 'failed') — call resume_training_job to continue from the last checkpoint.
1253
- Call get_training_metrics for semantic summaries first; request a bounded series when you need to inspect curve shape. Follow each metric's returned goal instead of assuming every number should decrease.
1249
+ ### Step 6: Monitor progress
1250
+ Call get_training_status for progress (queued, getting ready, downloading the model, preparing data, training step N of M, saving). Call get_training_metrics for loss, eval loss, gradient norm, token accuracy, learning rate and tokens trained. The job keeps two checkpoints: the best (lowest validation loss) and the last.
1251
+ If a started job loses its machine it rests at 'interrupted' (nothing is charged for that) -- call resume_training_job to continue from the last checkpoint.
1254
1252
 
1255
1253
  ### Tips
1256
1254
  - Start with LoRA, not full fine-tune — it's 10x faster and 5x cheaper
@@ -1357,25 +1355,21 @@ When to use: Before SFT, when your domain has specialized vocabulary/knowledge.
1357
1355
  - A VLM architecture does not imply support for image-only pretraining or multimodal preference objectives.
1358
1356
  - DPO, PPO and GRPO are not enabled by these training tools. They require separately verified training implementations; PPO/GRPO also require rollout inference and reward handling.
1359
1357
  - Training starts from catalog models. Do not promise a CPT-to-SFT pipeline using a customer's trained checkpoint as a new training source: that workflow is not exposed here.`,
1360
- gpu_selection: `# GPU Selection Guide
1361
-
1362
- ## Configure first, choose the GPU last
1363
- 1. Select the catalog model and revision, method, adapter and compatible datasets.
1364
- 2. Set supported configuration fields from get_training_capabilities: sequence length, batch and accumulation, validation and run-length bounds. Review effective precision/optimizer settings; leave disabled engine-managed controls such as activation checkpointing to the engine.
1365
- 3. The current preflight API requires a concrete gpu_type/gpu_count. Use get_recommended_gpu only to obtain a provisional candidate, then call preflight_training_job with that candidate and the complete configuration. Never treat default-config sizing as approval of the final run.
1366
- 4. Review the service-calculated minimum, supported counts and current quote. Choose only a qualifying available type/count. Fit is not stock, and an unknown availability result is not permission to book.
1367
- 5. Re-run preflight after changing any sizing input or GPU choice; create only after valid=true with the final selection. GPU-free preflight discovery is not currently supported by the deployed API.
1368
-
1369
- ## Booking and recovery
1370
- - Create rechecks capacity. Do not report a run as started while it is only securing; booked means capacity was accepted, not that the trainer has begun executing steps.
1371
- - A capacity miss must show fresh qualifying alternatives and preserve the configuration. Ask before substituting hardware or exceeding a price cap.
1372
- - Queueing is explicit consent and applies only when no qualifying alternative is available. Waiting is unbilled.
1373
- - Reuse the idempotency key after timeouts instead of creating a second run.
1374
-
1375
- ## Memory and quality
1376
- - Memory depends on the full configuration and input shapes, not parameter count alone. Do not invent a smaller minimum or claim that quantization always halves total memory.
1377
- - Full fine-tuning is not a guarantee of higher accuracy than adapters. Use representative validation to compare outcomes.
1378
- - Preflight admission is not proof of universal freedom from hardware faults or OOM. Report actual tested configurations and measured headroom.`,
1358
+ gpu_selection: `# Machines (managed training)
1359
+
1360
+ The platform chooses the machines. Do not choose, ask for, or describe GPUs, GPU counts, batch size, gradient accumulation, memory settings, checkpoint counts or price caps: the training tools do not take them, and the API ignores them with a FIELD_IGNORED warning.
1361
+
1362
+ ## How a job is placed
1363
+ - Every offered combination of model, adapter, context length, precision and optimizer was measured on our machines first. get_managed_training_options lists exactly what is offered for a model and adapter; anything else is refused before any money is spent, with a reason.
1364
+ - The platform picks the fastest tested setup that fits the job, with batch and memory settings that keep it from running out of memory, and checks the run's memory again before it starts.
1365
+ - Short samples are packed together into each context window, so a longer context costs no speed.
1366
+
1367
+ ## When no machine is free
1368
+ - The job waits in the queue, first come first served with fair turns per user. Waiting is never charged and a queued request expires after seven days.
1369
+ - It starts automatically and the user is emailed when it does. It can be cancelled at any time, queued or running.
1370
+
1371
+ ## What the user pays
1372
+ - Only for the exact tokens trained, at the model's per-token price. preflight_training_job returns the estimate.`,
1379
1373
  serverless_inference: `# Serverless Inference
1380
1374
 
1381
1375
  Serverless is the immediate, per-token inference product. It does NOT create or manage a deployment and does not reserve a GPU.
@@ -1490,149 +1484,55 @@ ignored rather than honored — the server derives them again either way.
1490
1484
  - Training and deployment use the shared capacity broker. It evaluates priority first, then rotates fairly across users at equal priority while preserving FIFO inside each user's own requests. This prevents a high-volume user from starving others; do not describe the queue as strict global FIFO.`,
1491
1485
  hyperparameters: `# Hyperparameter Guide
1492
1486
 
1493
- ## Key Hyperparameters
1494
-
1495
- ### Learning Rate
1496
- - Default: 2e-4 (0.0002)
1497
- - Range: 1e-5 to 5e-4
1498
- - Higher = faster learning but risk of instability
1499
- - Lower = more stable but slower convergence
1500
- - Tip: Start with 2e-4 for LoRA, 2e-5 for full fine-tune
1501
-
1502
- ### Batch Size (per_device_train_batch_size)
1503
- - Default: 4
1504
- - Range: 1-32 (limited by GPU VRAM)
1505
- - Larger = smoother gradients, more stable training
1506
- - Smaller = faster per-step but noisier
1507
- - Use gradient_accumulation_steps to simulate larger batches
1508
-
1509
- ### Epochs (num_train_epochs)
1510
- - Default: 3
1511
- - Range: 1-10
1512
- - More epochs = more passes through data
1513
- - Watch for overfitting: if eval loss starts increasing, you've trained too long
1514
- - For small datasets (<1000 rows): 3-5 epochs
1515
- - For large datasets (>10000 rows): 1-3 epochs
1516
-
1517
- ### LoRA Rank (lora_r)
1518
- - Default: 16
1519
- - Range: 4-256
1520
- - Higher = more trainable parameters, better quality, but slower
1521
- - 8-16 is sufficient for most tasks
1522
- - 32-64 for complex tasks requiring more model capacity
1523
- - 128+ for near-full-fine-tune quality
1524
-
1525
- ### LoRA Alpha (lora_alpha)
1526
- - Default: 32 (typically 2x lora_rank)
1527
- - Controls the scaling of LoRA updates
1528
- - Rule of thumb: set to 2x your lora_rank
1529
-
1530
- ### Max Sequence Length
1531
- - Default: 2048
1532
- - Range: 512-8192
1533
- - Must cover your longest training example
1534
- - Longer = more VRAM usage
1535
- - Set to slightly above your longest example
1536
-
1537
- ### LR Scheduler
1538
- - cosine (default) — gradually decreases, good for most tasks
1539
- - linear — constant decrease, predictable
1540
- - constant_with_warmup — stays flat after warmup, good for short training
1541
-
1542
- ### Warmup Ratio
1543
- - Default: 0.03 (3% of total steps)
1544
- - Prevents early instability by slowly ramping up learning rate
1545
-
1546
- ## Recommended Defaults by Use Case
1547
- | Parameter | Quick Test | Standard | High Quality |
1548
- |-----------|-----------|----------|--------------|
1549
- | Adapter | LoRA | LoRA | Full |
1550
- | Learning Rate | 3e-4 | 2e-4 | 2e-5 |
1551
- | Epochs | 1 | 3 | 5 |
1552
- | Batch Size | 4 | 4 | 8 |
1553
- | LoRA Rank | 8 | 16 | 64 |
1554
- | Scheduler | cosine | cosine | cosine |`,
1487
+ Offer only the settings below. Ranges and the context lengths, precisions and optimizers a model offers come from get_training_capabilities and get_managed_training_options.
1488
+
1489
+ ### Learning rate (learning_rate)
1490
+ How big a step the model takes each time it learns. Start with 2e-4 for LoRA and QLoRA and 1e-5 to 2e-5 for full fine-tuning. Lower it if the loss jumps around.
1491
+
1492
+ ### Epochs (epochs) and max steps (max_steps)
1493
+ How many times the model reads the whole dataset. One to three is usual: small datasets (under 1,000 rows) 3 to 5, large datasets 1 to 3. max_steps is an optional cap; leave it empty to train every epoch.
1494
+
1495
+ ### Context length (max_length)
1496
+ The longest text read in one go, in tokens: 2K, 4K, 8K, 16K, 32K, 64K or 128K where the model offers it. Pick one that fits the longest examples. Longer samples lose their oldest turns, never the answer.
1497
+
1498
+ ### Precision (precision)
1499
+ bf16 is accurate and fast and right for most runs. fp8 can be faster on large models and is offered only where it was tested.
1500
+
1501
+ ### Optimizer state (optim)
1502
+ adamw_torch_fused keeps 16-bit state and is the most accurate. adamw_8bit uses less memory, which helps large full fine-tunes, and is offered only where it was tested.
1503
+
1504
+ ### LoRA rank (lora_rank) and alpha (lora_alpha)
1505
+ Rank is how much the adapter can learn (8 to 64; 16 is a good start). Alpha is usually twice the rank.
1506
+
1507
+ ### Warmup ratio, weight decay and schedule
1508
+ warmup_ratio (default 0.03) eases the first steps in. weight_decay (around 0.01) helps the model generalize. lr_scheduler_type defaults to cosine.
1509
+
1510
+ ### Validation rows (split_dataset_samples) and early stopping (early_stopping_patience)
1511
+ Rows held out and never trained on, used to measure real progress and keep the best checkpoint. Allowed: 100 rows (or 5% when that is fewer) up to 40% of the rows; default 5% capped at 1,000. Early stopping ends the run after that many evaluations without improvement (default 3).`,
1555
1512
  monitoring: `# Training Monitoring Guide
1556
1513
 
1557
- ## Key Metrics to Watch
1558
-
1559
- ### Training Loss
1560
- - Should decrease steadily during training
1561
- - Sharp drops early are normal (learning basic patterns)
1562
- - Gradual decrease later shows fine-grained learning
1563
- - If loss plateaus: try higher learning rate or more LoRA rank
1564
- - If loss spikes: learning rate is too high, reduce it
1565
-
1566
- ### Eval Loss (if eval dataset provided)
1567
- - Should track training loss but may be slightly higher
1568
- - If eval loss increases while training loss decreases = OVERFITTING
1569
- - Early sign of overfitting: gap between train and eval loss growing
1570
- - Action: Stop training and use the last checkpoint before divergence
1571
-
1572
- ### Learning Rate
1573
- - Should follow your chosen schedule (cosine, linear, etc.)
1574
- - Warmup period: LR starts low and ramps up
1575
- - Main phase: LR follows the schedule downward
1576
-
1577
- ## Monitoring Tools
1578
- 1. get_training_status — Progress %, current step, elapsed time, ETA
1579
- 2. get_training_metrics — Semantic metric summaries plus bounded curve data on request
1580
- 3. get_training_logs — Raw training output for debugging
1581
- 4. get_training_checkpoints — Saved model snapshots at intervals
1582
-
1583
- ## When to Stop Early
1584
- - Loss has plateaued for many steps (no improvement)
1585
- - Eval loss is increasing (overfitting)
1586
- - You've reached satisfactory quality and want to save GPU cost
1587
- - The model is producing good outputs in eval samples
1588
-
1589
- ## Checkpoints
1590
- - Saved automatically at regular intervals during training
1591
- - Each checkpoint is a complete model snapshot you can use
1592
- - If training is interrupted, resume from the last checkpoint
1593
- - Best practice: compare outputs from different checkpoints to pick the best one`,
1594
- cost_optimization: `# Cost Optimization Guide
1595
-
1596
- ## Per-Second Billing
1597
- Run BiOS charges per-SECOND of GPU usage, not per hour. This means:
1598
- - You only pay for actual compute time
1599
- - Stopping a job immediately stops billing
1600
- - No wasted time rounding up to the next hour
1601
-
1602
- ## Cost Estimation
1603
- Before launching a job, estimate cost:
1604
- 1. Call get_recommended_gpu for your model — it includes a cost estimate
1605
- 2. Rough formula: (dataset_rows / batch_size) × epochs × seconds_per_step × gpu_price_per_second
1606
-
1607
- ## Money-Saving Strategies
1608
-
1609
- ### 1. Start Small
1610
- - Train for 1 epoch first to validate your setup
1611
- - Check if the loss is decreasing and outputs look reasonable
1612
- - Only then train for the full 3-5 epochs
1613
-
1614
- ### 2. Use LoRA
1615
- - LoRA is 5-10x cheaper than full fine-tune
1616
- - Quality is comparable for most tasks
1617
- - QLoRA is even cheaper (uses 4-bit quantization)
1618
-
1619
- ### 3. Optimize Batch Size
1620
- - Larger batch sizes = fewer total steps = less time
1621
- - Use gradient accumulation to simulate large batches on limited VRAM
1622
-
1623
- ### 4. Choose the Right GPU
1624
- - Don't pick H100 for a 7B LoRA job — A100 is plenty and cheaper
1625
- - Use get_recommended_gpu to find the cost-optimal choice
1626
-
1627
- ### 5. Monitor and Stop Early
1628
- - Read the summaries from get_training_metrics, then request a bounded series for any curve that needs inspection
1629
- - Treat a plateau as evidence to review alongside eval metrics and the configured run bound, not an automatic stop verdict
1630
- - Check wallet balance with get_wallet_balance before long jobs
1631
-
1632
- ### 6. Check Your Balance First
1633
- - Call get_wallet_balance before creating a job
1634
- - Ensure you have enough credits for the estimated training time
1635
- - Set up auto top-up to avoid interruptions mid-training`,
1514
+ ## Tools
1515
+ 1. get_training_status -- where the job is: queued, getting ready, downloading the model, preparing data, training step N of M, saving, finished.
1516
+ 2. get_training_metrics -- training metrics only: loss, eval loss, gradient norm, token accuracy, perplexity, learning rate, epoch and tokens trained. Read the summaries first and ask for a bounded series when you need the curve.
1517
+ 3. get_training_checkpoints -- the kept checkpoints: the best (lowest validation loss) and the last.
1518
+
1519
+ Training logs and machine metrics (memory, MFU, throughput) are not available to customers.
1520
+
1521
+ ## Reading the curves
1522
+ - Training loss should fall. A sharp early drop is normal.
1523
+ - Eval loss should follow it. If eval loss rises while training loss falls the model is starting to memorize; early stopping ends the run and the best checkpoint is kept.
1524
+ - The learning rate follows the schedule: warmup, then the decay.`,
1525
+ cost_optimization: `# Cost Guide
1526
+
1527
+ ## Per-token pricing
1528
+ Training is charged for the exact tokens it trained, at the model's per-token price for the adapter. Waiting in the queue, setup and machine time are never charged. preflight_training_job returns the estimate; the charge after the run is the exact count.
1529
+
1530
+ ## Spending less
1531
+ - Train 1 epoch first to check the setup, then the full run.
1532
+ - LoRA and QLoRA cost less per token than full fine-tuning and are close in quality for most tasks.
1533
+ - Early stopping (on by default when validation rows are held out) ends a run once it stops improving.
1534
+ - max_steps caps a run when you only want part of the data.
1535
+ - Check get_wallet_balance before long runs.`,
1636
1536
  capacity_errors: `# Standard GPU Rejection Errors
1637
1537
 
1638
1538
  Training and inference speak ONE GPU rejection contract: one body shape, and a
@@ -2559,6 +2459,21 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2559
2459
  return json(stripBuildIdentity(data));
2560
2460
  });
2561
2461
  /* ══════════════════════════════════════════════════════════════════════════ */
2462
+ /* TOOL: get_managed_training_options */
2463
+ /* ══════════════════════════════════════════════════════════════════════════ */
2464
+ server.tool("get_managed_training_options", "Managed training only (get_training_capabilities lists managed_training enabled): what this model, method and adapter can be trained with. Returns options, each a precision (bf16 or fp8) and optimizer state (adamw_torch_fused = 16-bit, adamw_8bit = 8-bit) with the context lengths (2048 ... 131072) a certified setup covers. Offer the customer only these; every option listed is accepted by preflight_training_job and create_training_job. Empty options means the model is not trainable with that method and adapter yet. No hardware is named and none should be described to the customer.", {
2465
+ model_id: z.string().describe("Model id, e.g. Qwen/Qwen3.5-9B."),
2466
+ method: z.enum(TRAINING_METHODS).optional().describe("Training method: sft (default), or pt/cpt for continued pre-training."),
2467
+ adapter: z.enum(TRAINING_ADAPTERS).optional().describe("lora (default), qlora or full."),
2468
+ }, async ({ model_id, method, adapter }) => {
2469
+ const q = new URLSearchParams({
2470
+ model_id,
2471
+ training_method: method === "cpt" ? "pt" : method || "sft",
2472
+ train_type: adapter || "lora",
2473
+ });
2474
+ return json(await client.api(`/api/training/managed-options?${q}`));
2475
+ });
2476
+ /* ══════════════════════════════════════════════════════════════════════════ */
2562
2477
  /* TOOL: preflight_training_job */
2563
2478
  /* ══════════════════════════════════════════════════════════════════════════ */
2564
2479
  /**
@@ -2602,7 +2517,7 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2602
2517
  ...(params.eval_dataset_ids ? { eval_dataset_ids: await Promise.all(params.eval_dataset_ids.map((ref) => resolveDatasetRef(ref, "eval_dataset_ids"))) } : {}),
2603
2518
  };
2604
2519
  }
2605
- server.tool("preflight_training_job", "Validate and canonicalize a proposed training job without creating a job, charging the wallet, entering the queue, or allocating a GPU. Returns dataset compatibility, model facts, authoritative GPU choices/prices, queue eligibility, alternatives, warnings, and a request hash.", {
2520
+ server.tool("preflight_training_job", "Validate and canonicalize a proposed training job without creating a job or charging the wallet. You choose the model, adapter, datasets and training settings; the platform chooses the machines, batch and memory settings from tested setups. Returns dataset compatibility, the per-token cost estimate, the validation rows it will hold out (and their allowed range), warnings (FIELD_IGNORED for anything the platform sets) and a request hash.", {
2606
2521
  model: z.string().describe("Base model id from the Run BiOS catalog (e.g., 'meta-llama/Llama-3.1-8B-Instruct'). Must be hosted on Run BiOS — pick from list_models/search_models. Training always starts from a catalog base model; start_from (where enabled) adds earlier LoRA versions of that same base on top of it, and training from any other fine-tuned model is not supported."),
2607
2522
  model_revision: z.string().max(256).optional().describe("Requested model branch, tag, or commit. Copy the exact 40-hex value returned in canonical_request into create."),
2608
2523
  dataset_id: z.string().optional().describe("Dataset id from list_datasets. An exact, unique dataset name is also accepted and resolved to its id."),
@@ -2610,25 +2525,7 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2610
2525
  eval_dataset_ids: z.array(z.string()).min(1).max(64).optional().describe("Separate evaluation datasets, composed sequentially. Mutually exclusive with eval_split_ratio."),
2611
2526
  method: z.enum(TRAINING_METHODS),
2612
2527
  adapter: z.enum(TRAINING_ADAPTERS).optional(),
2613
- gpu_type: z.string().optional(),
2614
- gpu_count: z.number().int().min(1).max(8).optional(),
2615
- gpu_priorities: z
2616
- .array(z.object({
2617
- gpu_type: z.string(), gpu_count: z.number().int().min(1).max(8),
2618
- provider: z.string().min(1).max(30).optional(), region: z.string().min(1).max(64).optional(),
2619
- tier: z.literal("secure").optional(),
2620
- }))
2621
- .min(1)
2622
- .max(5)
2623
- .optional()
2624
- .describe("Ranked GPU placement choices. Copy these unchanged from preflight; queueing accepts one exact placement through five ranked choices, and entry one must match gpu_type/gpu_count."),
2625
- queue_if_unavailable: z.boolean().optional().describe("Explicit consent to wait in the capacity queue when none of the ranked GPU choices are free; accepts 1 to 5 gpu_priorities because some training configurations have one valid shape."),
2626
- queue_deadline: z.string().min(1).optional().describe("Optional RFC 3339 deadline from one minute through seven days in the future; requires queue_if_unavailable=true."),
2627
- max_price_hour_cents: priceCapSchema(true),
2628
- storage_gb: z.number().int().min(1).max(10000).optional(),
2629
- num_checkpoints: z.number().int().min(1).max(100).optional(),
2630
2528
  integration_id: z.string().optional(),
2631
- network_volume_id: z.string().optional(),
2632
2529
  dataset_mixing: z.enum(["shuffle", "sequential", "interleave"]).optional(),
2633
2530
  mixing: datasetMixingSchema.optional().describe("Composition plan. Dataset IDs define source order and weight positions; default seed is 42. The source pins, plan and checksum are retained for metadata-only resume."),
2634
2531
  start_from: startFromSchema.optional(),
@@ -2649,7 +2546,7 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2649
2546
  // server-side (403 TRAINING_COMING_SOON before any billing or booking) as the
2650
2547
  // authority. At launch the flag opens and the registration returns verbatim.
2651
2548
  if (!launchGates.training) {
2652
- server.tool("create_training_job", "Create a training job using the same BOOK-BEFORE-REVEAL contract as the web UI. Call preflight_training_job first with the complete model/method/adapter/dataset/config request, then choose one returned GPU configuration: GPU is last because its minimum depends on every earlier choice. The call does not reveal a run as started until a real pod is secured; status is booked (secured) or securing (poll get_training_status). A booking-time miss returns CAPACITY_UNAVAILABLE with fresh qualifying available_gpus and no job/no charge: choose one and retry with the same setup. The rejected GPU type is never recommended back. Queue consent is valid only when available_gpus is empty; waiting is unbilled and expires automatically. Reuse one idempotency key after timeouts.", {
2549
+ server.tool("create_training_job", "Create a training job with the same request preflight_training_job validated. The platform places it on tested machines; when none is free it waits in the queue (first come first served, unbilled, expires after seven days), starts automatically and emails the user when it starts. It is charged for the exact tokens it trained. Report progress from get_training_status and training metrics from get_training_metrics; cancel any time with stop_training_job. Reuse one idempotency key after timeouts.", {
2653
2550
  model: z.string().describe("Base model id from the Run BiOS catalog (e.g., 'meta-llama/Llama-3.1-8B-Instruct'). Must be hosted on Run BiOS — pick from list_models/search_models. Training always starts from a catalog base model; start_from (where enabled) adds earlier LoRA versions of that same base on top of it, and training from any other fine-tuned model is not supported."),
2654
2551
  model_revision: z.string().max(256).optional().describe("Exact model_revision from preflight canonical_request; a branch or tag is accepted but resolved again before any paid mutation."),
2655
2552
  idempotency_key: z
@@ -2664,25 +2561,7 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2664
2561
  eval_dataset_ids: z.array(z.string()).min(1).max(64).optional().describe("Separate evaluation datasets, composed sequentially. Mutually exclusive with eval_split_ratio."),
2665
2562
  method: z.enum(TRAINING_METHODS).describe("Training method: sft, or pt/cpt for continued pre-training."),
2666
2563
  adapter: z.enum(TRAINING_ADAPTERS).optional().describe("Any adapter supported by the Run BiOS contract."),
2667
- gpu_type: z.string().optional().describe("GPU type returned by preflight_training_job after the complete training configuration is known. Do not guess or use a pre-configuration recommendation."),
2668
- gpu_count: z.number().int().min(1).max(8).optional().describe("Number of GPUs for the primary choice."),
2669
- gpu_priorities: z
2670
- .array(z.object({
2671
- gpu_type: z.string(), gpu_count: z.number().int().min(1).max(8),
2672
- provider: z.string().min(1).max(30).optional(), region: z.string().min(1).max(64).optional(),
2673
- tier: z.literal("secure").optional(),
2674
- }))
2675
- .min(1)
2676
- .max(5)
2677
- .optional()
2678
- .describe("Ranked GPU placement choices. Copy these unchanged from preflight; queueing accepts one exact placement through five ranked choices, and entry one must match gpu_type/gpu_count."),
2679
- queue_if_unavailable: z.boolean().optional().describe("Explicit consent to wait in the capacity queue when none of the ranked GPU choices are free; accepts 1 to 5 gpu_priorities because some training configurations have one valid shape."),
2680
- queue_deadline: z.string().min(1).optional().describe("Optional RFC 3339 deadline from one minute through seven days in the future; requires queue_if_unavailable=true."),
2681
- max_price_hour_cents: priceCapSchema(true),
2682
- storage_gb: z.number().int().min(1).max(10000).optional(),
2683
- num_checkpoints: z.number().int().min(1).max(100).optional(),
2684
2564
  integration_id: z.string().optional(),
2685
- network_volume_id: z.string().optional(),
2686
2565
  dataset_mixing: z.enum(["shuffle", "sequential", "interleave"]).optional(),
2687
2566
  mixing: datasetMixingSchema.optional().describe("Composition plan. Dataset IDs define source order and weight positions; default seed is 42. The source pins, plan and checksum are retained for metadata-only resume."),
2688
2567
  start_from: startFromSchema.optional(),
@@ -2766,7 +2645,7 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2766
2645
  /* ══════════════════════════════════════════════════════════════════════════ */
2767
2646
  /* TOOL: get_training_logs */
2768
2647
  /* ══════════════════════════════════════════════════════════════════════════ */
2769
- server.tool("get_training_logs", "Get structured recent training log entries (level, message, timestamp). Results are newest first and bounded by tail.", {
2648
+ server.tool("get_training_logs", "Get structured recent training log entries (level, message, timestamp). Results are newest first and bounded by tail. Not available when managed_training is enabled (the platform runs the machines): use get_training_status for progress and get_training_metrics for loss, eval loss, gradient norm and accuracy.", {
2770
2649
  job_id: z.string().describe("Training job ID"),
2771
2650
  tail: z
2772
2651
  .number()
@@ -2777,9 +2656,21 @@ create (1-5 choices), and the queue itself stays opt-in.`,
2777
2656
  .describe("Number of most recent log entries to return (default: 200, max: 1000)"),
2778
2657
  }, async ({ job_id, tail }) => {
2779
2658
  validateId(job_id, "job_id");
2780
- const data = await client.api(`/api/training/jobs/${job_id}/logs`, {
2781
- params: { tail: tail?.toString() },
2782
- });
2659
+ let data;
2660
+ try {
2661
+ data = await client.api(`/api/training/jobs/${job_id}/logs`, {
2662
+ params: { tail: tail?.toString() },
2663
+ });
2664
+ }
2665
+ catch (error) {
2666
+ if (/\bLOGS_NOT_AVAILABLE\b/.test(error instanceof Error ? error.message : String(error))) {
2667
+ return json({
2668
+ logs: [],
2669
+ note: "Training logs are not available under managed training. Use get_training_status for progress and get_training_metrics for training metrics.",
2670
+ });
2671
+ }
2672
+ throw error;
2673
+ }
2783
2674
  // Log lines are written by the training engine, so a boot/pull line can
2784
2675
  // quote the image reference it started from. Redact by content: the
2785
2676
  // customer keeps every line, minus any build identity inside it.
@@ -3428,6 +3319,11 @@ create (1-5 choices), and the queue itself stays opt-in.`,
3428
3319
  method: "POST",
3429
3320
  body: {},
3430
3321
  })));
3322
+ server.tool("loop_list_live_deployments", "For a pipeline whose live version's deployment was DELETED (live_deployment.status deleted on loop_get_pipeline): the deployments of this workspace that serve exactly the live version's weights, with their status, and the live version itself (live.training_job_id and live.checkpoint_id, what the user deploys again from Deployments). Nothing has to be trained again: the weights are kept. 409 NO_LIVE_VERSION when the pipeline has none. Read-only.", { pipeline_id: z.string() }, async ({ pipeline_id }) => json(await client.api(`/api/loop/pipelines/${encodeURIComponent(pipeline_id)}/live/deployments`)));
3323
+ server.tool("loop_use_live_deployment", "Answer the pipeline's model name from this deployment of the live version's weights (one loop_list_live_deployments named), after the pipeline's own deployment was deleted. Nothing is trained again and the version number does not change; the deployment bills the user by the hour, like any deployment, so say so. Refused while the pipeline's own deployment still exists (409 LIVE_DEPLOYMENT_PRESENT: the user starts it again in Deployments instead), for other weights (422 DEPLOYMENT_WEIGHTS_DIFFER) and for a deployment that is not running (409 DEPLOYMENT_NOT_RUNNING). A later rollback stops it.", { pipeline_id: z.string(), deployment_id: z.string() }, async ({ pipeline_id, deployment_id }) => json(await client.api(`/api/loop/pipelines/${encodeURIComponent(pipeline_id)}/live/deployment`, {
3324
+ method: "POST",
3325
+ body: { deployment_id },
3326
+ })));
3431
3327
  server.tool("loop_import_into_pipeline", "loop_import_rows with every row given this pipeline's ID (own_tag), so the file becomes this pipeline's samples. source defaults to the pipeline ID. default_feedback 'good' records every answered row that brought no feedback of its own as a good example to learn from — use it only when the user says the file's answers are good; without it such rows wait for review. The result, the retry token (import_id) and the 5000-row limit are loop_import_rows' own.", {
3432
3328
  pipeline_id: z.string(),
3433
3329
  rows: z.array(z.record(z.string(), z.any())).describe("The rows, already parsed, at most 5000, each in the shape it already has"),