flashruntime 0.3.0__py3-none-any.whl

flashruntime/viewer/_docs/search-index.json

@@ -0,0 +1 @@
+[{"url": "index.html", "title": "FlashRuntime", "text": "FlashRuntime FlashRuntime operates your training job — it never rewrites it. You keep the model, the training loop, the loss, the data, and the framework you already have. FlashRuntime wraps the reliability and reproducibility layer around them: it launches your command, injects the environment it promised, tracks your metrics, validates your checkpoints, retries on failure, and collects your artifacts. You own FlashRuntime operates model, training loop, loss, data, framework launch, environment, metric tracking, checkpoint validity, recovery, artifact collection The contract at the boundary is deliberately thin: arguments in, metrics.json out. A script that already reads its hyperparameters from argparse and writes a small JSON file of results needs zero FlashRuntime imports to be operated. This is ADR-0003 's fourth axis in practice: recipes integrate user code; the distributed math is always done by your framework (PyTorch DDP, torchrun , Hugging Face, sklearn). The 60-second demo Install it, point it at a script, and watch it run — recovering across crashes on the way: Copy import flashruntime as flash run = flash.submit( flash.CommandWorkload( command=\"python train.py --epochs 5\", source=flash.Source(path=\"~/my-project\"), outputs=flash.OutputSpec(collect=[\"metrics.json\"]), ), max_restarts=2, # a crashed attempt is classified, then relaunched from # the last VALID checkpoint — up to twice watch=True, # opens the live run page and prints its URL ) print(run.state.value) # \"SUCCEEDED\" (or \"FAILED\") print(run.artifacts) # [PosixPath('.../metrics.json'), ...] print(run.viewer_url) # http://127.0.0.1:<port> — the live run page flash.submit() compiles that description into a launch spec, runs it as a real subprocess, waits, and hands back a Run . command is shlex -split (there is no shell — for a pipe, pass command=\"bash -c '...'\" ), and source is a flash.Source , so ~ is expanded for you. max_restarts is the automatic fault-tolerance budget. On a FAILED attempt FlashRuntime turns the exit into failure signals, classifies them, and consults a versioned, deterministic recovery policy: a deterministic application bug fails fast (a retry only re-hits it); anything else relaunches the same spec from the job-scoped checkpoint, up to the budget. Same failure same policy version ⇒ same action, every time — no LLM in the loop. watch=True opens the live run page in a browser (and records its URL on run.viewer_url ). It draws the run's topology, its loss curve, its verified checkpoints, and every recovery decision, polling a loopback server with zero external assets . watch defaults to auto: on at an interactive terminal, off in a pipe or CI. These docs are served by the same local server at /docs . What FlashRuntime does around your job Launch — starts your command as a subprocess (locally today; leased to a remote node through the coordinator when you compile it to a JobSpec). Environment — injects the env vars a run promises, so the same command is reproducible across machines. Tracking — reads the metrics.json your script writes and records it as a trial; a fan-out sweep merges each trial's parameters. Checkpoints — the parts-first / manifest-last contract means a half-written checkpoint can never look valid; recovery restores only a verified, topology-compatible manifest. Recovery — typed, deterministic, logged. Every retry emits a FAILURE_CLASSIFIED and a RECOVERY_ACTION_SELECTED event carrying the failure class and the policy's human-readable reason. Artifacts — copies your outputs.collect globs out of the run before the next trial can overwrite them. Next Head to Get started to install FlashRuntime, run your first job, and launch your first 2-process DDP run on CPU — no cluster, no GPU required."}, {"url": "get-started.html", "title": "Get started", "text": "Get started This page takes you from an empty environment to a fault-tolerant job and your first 2-process DDP run — on CPU, with no cluster and no GPU. For the what and why, see the overview . Install The core is deliberately tiny — pip install flashruntime brings only pydantic , and every core module (planner, leases, checkpoints, recovery, the flash.submit() SDK) works with zero infrastructure: Copy pip install flashruntime Infrastructure integrations are opt-in extras, never core imports: Copy pip install \"flashruntime[service]\" # the FastAPI coordinator + CLI pip install \"flashruntime[sklearn]\" # numpy + scikit-learn for the sweep examples Torch is not a dependency. FlashRuntime launches PyTorch; it never imports it (the four-axes rule — launching is orthogonal to your framework). To run the DDP example below, install PyTorch yourself. A CPU-only build is enough — DDP works over the gloo backend with no GPU: Copy pip install torch # CPU build is fine; gloo needs no CUDA Your first run flash.submit() operates any command. The one convention your script owes FlashRuntime is to write a metrics.json (a flat JSON object) into its working directory; FlashRuntime collects it and records it as a trial: Copy import flashruntime as flash run = flash.submit(flash.CommandWorkload( command=\"python -c \\\"import json; json.dump({'accuracy': 0.91}, open('metrics.json','w'))\\\"\", source=flash.Source(path=\".\"), )) print(run.state.value) # \"SUCCEEDED\" print(run.trials) # [{'accuracy': 0.91}] print(run.artifacts) # [PosixPath('.../metrics.json')] Rerunning flash.submit(workload, output_dir=...) against the same output_dir reuses the job id, so a checkpointed script resumes instead of restarting. Pair that with max_restarts=N and a crash mid-run is recovered automatically from the last valid checkpoint. Your first DDP run The integrations.pytorch adapter builds the torchrun command for you. If your script already calls torch.distributed.init_process_group() and wraps its model in DistributedDataParallel , there are zero code changes — the adapter just launches it: Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit(fr_torch.ddp( \"train.py\", source=\"examples/user_pytorch_vanilla\", nproc_per_node=2, # 2 processes on this host — gloo/CPU works script_args=\"--steps 100\", )) print(run.state.value, run.trials) ddp(script, *, source=\".\", nproc_per_node=2, nnodes=1, script_args=\"\", env=None) emits torchrun --nproc-per-node=N --nnodes=1 --standalone --local-addr=127.0.0.1 <script> <args> . The --local-addr=127.0.0.1 pins the advertised rendezvous address to loopback (otherwise torchrun advertises socket.getfqdn() , which on some macOS DNS setups is unresolvable and the run hangs before spawning a process). nproc_per_node=2 starts two worker processes that rendezvous on loopback and hand each rank its RANK / WORLD_SIZE / LOCAL_RANK — a real distributed run on a single machine, no GPU required. nnodes > 1 raises NotImplementedError today: multi-node rendezvous is a launcher concern for a later slice. --standalone is single-node by definition. Want fault-tolerant checkpointing inside a script you are willing to touch? import flashruntime.torch as ft gives you ft.prepare(...) , ft.checkpoint(...) , and ft.log_metrics(...) — torch's own DDP wrapped under the parts-first / manifest-last checkpoint contract, so a killed run resumes with its final loss matching an uninterrupted run to 1e-6 (the e2e's assertion). It is optional sugar on the same launch-only contract, never required. The same flashruntime.torch path runs unchanged on GPUs — ft.prepare places each rank's model on its cuda:N , initializes the nccl backend, and restores checkpoints across the CUDA↔CPU boundary. Validated on real GPUs (2×RTX 4090, nccl) — 2026-07-23 (torch 2.7.1+cu128, CUDA 12.8), covering the 2-process nccl DDP run and a GPU kill-and-resume; see tests/test_gpu_e2e.py . Watch it run Pass watch=True (or just run at an interactive terminal, where it is the default) and flash.submit() opens a live run page in your browser and prints its URL: Copy import flashruntime as flash run = flash.submit( flash.CommandWorkload(command=\"python train.py\", source=flash.Source(path=\".\")), watch=True, ) print(run.viewer_url) # http://127.0.0.1:<port> The page draws the run's topology, loss curve, verified checkpoints, and every recovery decision, refreshing every couple of seconds — served entirely from a loopback server with no external assets , so it renders with the network cut. These docs are served from that same viewer at /docs ."}, {"url": "tutorials/convnet.html", "title": "Tutorial: make a ConvNet fault-tolerant", "text": "Tutorial: make a ConvNet fault-tolerant This is the flagship walkthrough. You start with an ordinary PyTorch ConvNet and an ordinary training loop, and end with the same code — unchanged in its math — running as a 2-process DDP job that survives a crash and resumes from its last verified checkpoint, with a live page you can watch. The promise throughout: FlashRuntime operates your code; it never rewrites it. The model below is used verbatim, including one unusual thing it does, because \"your code, unmodified\" is the whole point. If you have not installed FlashRuntime and PyTorch yet, do the Get started page first — you need pip install flashruntime and a CPU build of torch . No GPU, no cluster. 1. The plain script (no FlashRuntime) Here is the model and a plain single-process training loop. It reads its hyperparameters from argparse and writes a metrics.json at the end — the one convention FlashRuntime asks of any script. There is no FlashRuntime import yet. Copy import argparse import json import torch import torch.nn as nn from torch.utils.data import DataLoader, TensorDataset class OurConvNet(torch.nn.Module): def __init__(self, num_outputs=20): super(OurConvNet, self).__init__() self.Conv1 = nn.Conv2d(3, 64, 5, 1, 2) self.Sigma = nn.Sigmoid() self.Avg = nn.AvgPool2d(2, stride=2, padding=0) self.Conv2 = nn.Conv2d(64, 128, 5, 1) self.Fl = nn.Flatten() self.Linear = nn.Linear(128 * 5 * 5, num_outputs) def forward(self, x): out = self.Sigma(x) # NOTE: the Sigmoid is applied to the INPUT first. out = self.Conv1(out) out = self.Avg(out) out = self.Conv2(out) out = self.Avg(out) out = self.Fl(out) out = self.Linear(out) return out def make_data(n=256, num_outputs=20, seed=0): # Synthetic data with a fixed seed keeps the run deterministic on CPU. g = torch.Generator().manual_seed(seed) x = torch.randn(n, 3, 28, 28, generator=g) y = torch.randint(0, num_outputs, (n,), generator=g) return TensorDataset(x, y) def main(): parser = argparse.ArgumentParser() parser.add_argument(\"--steps\", type=int, default=40) parser.add_argument(\"--lr\", type=float, default=0.05) args = parser.parse_args() torch.manual_seed(0) model = OurConvNet(num_outputs=20) optimizer = torch.optim.SGD(model.parameters(), lr=args.lr) loader = DataLoader(make_data(), batch_size=32, shuffle=False) step = 0 loss = torch.tensor(0.0) while step < args.steps: for x, y in loader: if step >= args.steps: break loss = torch.nn.functional.cross_entropy(model(x), y) optimizer.zero_grad() loss.backward() optimizer.step() step += 1 with open(\"metrics.json\", \"w\") as f: json.dump({\"steps\": step, \"final_loss\": round(loss.item(), 6)}, f) print(\"done\", step, loss.item()) if __name__ == \"__main__\": main() Run it like any script: Copy python train.py --steps 30 The shape math (so 128 * 5 * 5 is not a magic number) The input is 3 × 28 × 28 . Follow one image through forward , and every nn.Linear(128 * 5 * 5, ...) factor falls out of the convolution arithmetic: Copy 28 --Conv1(k5, s1, p2)--> 28 (padding 2 keeps the size) --Avg(2, 2)----------> 14 --Conv2(k5, s1, p0)--> 10 (no padding: 14 - 5 + 1 = 10) --Avg(2, 2)----------> 5 => Flatten = 128 channels · 5 · 5 = 3200 = nn.Linear(128 * 5 * 5, num_outputs) So 3 × 28 × 28 is the input size that makes the flattened feature map land exactly on 128 * 5 * 5 . Feed a different size and the Flatten → Linear handoff mismatches; keep 28 and it fits. One honest note about this model forward applies self.Sigma (a Sigmoid ) to the raw input before the first convolution — an unusual ordering (a sigmoid is normally an activation between layers, not a preprocessing step on the pixels). We keep it exactly as written. FlashRuntime's promise is to operate your code, so we do not \"fix\" the model to match convention — we run what you wrote. 2. Make it resumable: import flashruntime.torch as ft The plain script has no checkpoints, so a crash starts over from step 0. One import — flashruntime.torch (aliased ft ) — gives you launch-anywhere DDP and fault-tolerant checkpointing without rebuilding any framework machinery. Here is the whole change, as a diff: Copy import argparse import json import torch import torch.nn as nn from torch.utils.data import DataLoader, TensorDataset +import flashruntime.torch as ft + def main(): parser = argparse.ArgumentParser() parser.add_argument(\"--steps\", type=int, default=40) parser.add_argument(\"--lr\", type=float, default=0.05) + parser.add_argument(\"--checkpoint-every\", type=int, default=8) + parser.add_argument(\"--kill-at-step\", type=int, default=None, + help=\"simulate a crash (fresh runs only; a resumed retry finishes)\") args = parser.parse_args() torch.manual_seed(0) model = OurConvNet(num_outputs=20) optimizer = torch.optim.SGD(model.parameters(), lr=args.lr) loader = DataLoader(make_data(), batch_size=32, shuffle=False) - step = 0 + model, optimizer, loader = ft.prepare(model, optimizer, loader) + start = ft.start_step() + + step = start loss = torch.tensor(0.0) while step < args.steps: for x, y in loader: if step >= args.steps: break loss = torch.nn.functional.cross_entropy(model(x), y) optimizer.zero_grad() loss.backward() optimizer.step() step += 1 + ft.checkpoint(model, optimizer, step=step, every=args.checkpoint_every) + ft.log_metrics({\"step\": step, \"loss\": round(loss.item(), 6)}) + if args.kill_at_step and start == 0 and step >= args.kill_at_step: + raise SystemExit(3) # fresh run only — the retry resumes past this - with open(\"metrics.json\", \"w\") as f: - json.dump({\"steps\": step, \"final_loss\": round(loss.item(), 6)}, f) - print(\"done\", step, loss.item()) + ft.checkpoint(model, optimizer, step=step) # final checkpoint + if ft.is_main(): + metrics = {\"steps\": step, \"resumed_from\": start, \"final_loss\": round(loss.item(), 6)} + with open(\"metrics.json\", \"w\") as f: + json.dump(metrics, f) + print(metrics) What each added call does — and nothing more (this is the whole surface): ft.prepare(model, optimizer, loader) — launched distributed ( WORLD_SIZE > 1 ), it initializes torch's own process group ( gloo on CPU, nccl on GPU), wraps the model in DistributedDataParallel , and swaps the DataLoader's sampler for a seed-0 DistributedSampler so each rank sees a disjoint shard. It then restores the newest valid checkpoint if one exists. Launched as plain python train.py , it is a no-op passthrough. ft.start_step() — 0 on a fresh run, >0 after a resume (it is the step prepare restored to). The loop starts from it. ft.checkpoint(model, optimizer, step=step, every=8) — rank 0 writes a checkpoint under the parts-first / manifest-last contract; the manifest is written last , so a half-written checkpoint is never treated as valid. ft.log_metrics({...}) — rank 0 appends one JSON line to metrics.jsonl ; this is the streaming series the live page draws as a loss curve. It never raises. The full surface is three verbs plus read-only launch-fact accessors — see the torch helper reference . There are no FSDP policies, no autocast, no DeepSpeed config here: that is deliberate (ADR-0003 — we do not rebuild Accelerate). A script that wants those uses the real framework features; the launcher still launches it correctly. The finished, copy-paste-runnable version of this script is in section 5 below. 3. Operate it: flash.submit(fr_torch.ddp(...)) Now hand the script to FlashRuntime. The integrations.pytorch adapter builds the torchrun command that starts N worker processes; flash.submit() runs it, waits, collects the artifacts, and hands back a Run : Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit(fr_torch.ddp( \"train.py\", source=\".\", # directory holding train.py nproc_per_node=2, # two worker processes on this host — gloo/CPU, no GPU script_args=\"--steps 16\", )) print(run.state.value, run.trials) # SUCCEEDED [{'steps': 16, 'resumed_from': 0, 'final_loss': 2.98...}] ddp(script, *, source=\".\", nproc_per_node=2, nnodes=1, script_args=\"\", env=None) emits torchrun --nproc-per-node=2 --nnodes=1 --standalone --local-addr=127.0.0.1 train.py --steps 16 . Two processes rendezvous on loopback; each gets its RANK / WORLD_SIZE / LOCAL_RANK , and ft.prepare() wires DDP from there. nnodes > 1 raises NotImplementedError today — multi-node rendezvous is a later slice. torchrun must be on your PATH . It ships with torch ; if you installed torch into a virtualenv, run from that environment so its torchrun is found. 4. Crash it, and let it recover Now the payoff. Point a run at a fixed output_dir so its checkpoint tree persists, tell the script to crash mid-way with --kill-at-step , and give submit() a restart budget with max_restarts=1 . That is the only change — one keyword argument — and recovery is automatic: Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit( fr_torch.ddp( \"train.py\", source=\".\", nproc_per_node=1, script_args=\"--steps 24 --checkpoint-every 8 --kill-at-step 8\", ), output_dir=\"out/convnet\", # a FIXED dir — the checkpoint tree lives here max_restarts=1, # one automatic recovery attempt ) print(run.state.value, run.trials) # SUCCEEDED [{'steps': 24, 'resumed_from': 8, 'final_loss': 2.99...}] resumed_from: 8 is the proof: the first attempt crashed at step 8, and the retry resumed from the step-8 checkpoint rather than restarting at zero. The run's event log tells the whole story: Copy LAUNCH_STARTED task-000 launched (pid ...) FAILURE_CLASSIFIED task-000: worker_crash (exit 1) RECOVERY_ACTION_SELECTED task-000: restart_group — a lost rank stops the group — restart all workers from latest valid checkpoint LAUNCH_STARTED task-000-r1 launched (pid ...) That decision is not a guess. FlashRuntime turned the crash into failure signals, classify() 'd them (a torchrun -wrapped worker death is a non-deterministic worker_crash , not a code bug), and looked the class up in a versioned, deterministic policy table. Same failure + same policy version ⇒ same action, every time — no LLM in the loop. The mechanics are the subject of the fault-tolerance tutorial . One honest constraint on bit-exact resume: on resume the for loop restarts the dataloader at batch 0, so the resumed step must land on an epoch boundary . With 256 samples / batch 32 = 8 batches per epoch single-process, keep --checkpoint-every (and --kill-at-step ) multiples of that. Off a boundary, the resumed run is still correct training — it just will not match an uninterrupted run byte-for-byte. Watch it live Pass watch=True (or just run at an interactive terminal, where it is the default) and flash.submit() opens a live run page in your browser and records its URL on run.viewer_url : Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit( fr_torch.ddp(\"train.py\", source=\".\", nproc_per_node=2, script_args=\"--steps 16\"), output_dir=\"out/convnet\", watch=True, ) print(run.viewer_url) # http://127.0.0.1:<port> The page polls a loopback server every couple of seconds and draws the run with zero external assets (no CDN, no web font, no remote image) — it renders with the network cut. Top to bottom, the panels are: Header — a colored state badge (RUNNING cyan, SUCCEEDED green, FAILED red), the exact command, the execution mode , restarts used out of your budget, and the total attempts count. Topology — a single machine box labeled 127.0.0.1 · localhost with one node per attempt/rank, colored by state; a RUNNING node softly pulses. A 2-process DDP run shows two nodes. Loss — an autoscaled curve built from the metrics.jsonl your ft.log_metrics(...) calls stream, with the latest value labeled. (No log_metrics calls ⇒ no curve — this panel is fed by that stream.) Checkpoints — one violet marker per manifest: its step, a hash-verified or invalid badge (re-verified at read time, not taken on trust), a ★ latest tag on the one recovery would restore, and the part count plus age. Events — newest first, the same log shown above; FAILURE_CLASSIFIED is amber and RECOVERY_ACTION_SELECTED is cyan, each carrying the failure class and the policy's human-readable reason. Logs — a collapsible tail of each attempt's captured stdout+stderr. These docs are served from that same viewer at /docs , so the page you watch and the page you are reading are one product. 5. The final script (copy-paste runnable) The complete train.py from sections 2–4. It runs three ways from one file — python train.py , torchrun ... train.py , or flash.submit(fr_torch.ddp( ...)) — and needs only a CPU: Copy \"\"\"ConvNet tutorial — your model, unmodified, made fault-tolerant. python train.py --steps 40 # single process torchrun --nproc-per-node=2 --standalone train.py # DDP by hand flash.submit(fr_torch.ddp(\"train.py\", ...)) # operated by FlashRuntime Shapes: input 3x28x28 -> Conv1(k5,p2) 28 -> Avg 14 -> Conv2(k5) 10 -> Avg 5, so Flatten = 128 * 5 * 5 = 3200 = nn.Linear(128*5*5, num_outputs). \"\"\" import argparse import json import torch import torch.nn as nn from torch.utils.data import DataLoader, TensorDataset import flashruntime.torch as ft class OurConvNet(torch.nn.Module): def __init__(self, num_outputs=20): super(OurConvNet, self).__init__() self.Conv1 = nn.Conv2d(3, 64, 5, 1, 2) self.Sigma = nn.Sigmoid() self.Avg = nn.AvgPool2d(2, stride=2, padding=0) self.Conv2 = nn.Conv2d(64, 128, 5, 1) self.Fl = nn.Flatten() self.Linear = nn.Linear(128 * 5 * 5, num_outputs) def forward(self, x): out = self.Sigma(x) # NOTE: Sigmoid applied to the INPUT first — kept verbatim. out = self.Conv1(out) out = self.Avg(out) out = self.Conv2(out) out = self.Avg(out) out = self.Fl(out) out = self.Linear(out) return out def make_data(n=256, num_outputs=20, seed=0): g = torch.Generator().manual_seed(seed) x = torch.randn(n, 3, 28, 28, generator=g) y = torch.randint(0, num_outputs, (n,), generator=g) return TensorDataset(x, y) def main(): parser = argparse.ArgumentParser() parser.add_argument(\"--steps\", type=int, default=40) parser.add_argument(\"--lr\", type=float, default=0.05) parser.add_argument(\"--checkpoint-every\", type=int, default=8) parser.add_argument(\"--kill-at-step\", type=int, default=None, help=\"simulate a crash (fresh runs only; a resumed retry finishes)\") args = parser.parse_args() torch.manual_seed(0) model = OurConvNet(num_outputs=20) optimizer = torch.optim.SGD(model.parameters(), lr=args.lr) loader = DataLoader(make_data(), batch_size=32, shuffle=False) model, optimizer, loader = ft.prepare(model, optimizer, loader) start = ft.start_step() step = start loss = torch.tensor(0.0) while step < args.steps: for x, y in loader: if step >= args.steps: break loss = torch.nn.functional.cross_entropy(model(x), y) optimizer.zero_grad() loss.backward() optimizer.step() step += 1 ft.checkpoint(model, optimizer, step=step, every=args.checkpoint_every) ft.log_metrics({\"step\": step, \"loss\": round(loss.item(), 6)}) if args.kill_at_step and start == 0 and step >= args.kill_at_step: raise SystemExit(3) # fresh run only — the retry resumes past this ft.checkpoint(model, optimizer, step=step) # final checkpoint if ft.is_main(): metrics = {\"steps\": step, \"resumed_from\": start, \"final_loss\": round(loss.item(), 6)} with open(\"metrics.json\", \"w\") as f: json.dump(metrics, f) print(metrics) if __name__ == \"__main__\": main() Where to go next Fault tolerance, in depth — how a crash becomes signals, a class, and a typed recovery action. PyTorch guide — the two launch paths (already-DDP scripts vs. the ft. helper) and every caveat. Architecture — the four axes, leases, manifests, and recovery that make the above work."}, {"url": "tutorials/sklearn-sweeps.html", "title": "Tutorial: parallel scikit-learn sweeps", "text": "Tutorial: parallel scikit-learn sweeps scikit-learn work is embarrassingly parallel across runs — a grid of independent fits, never a single .fit() you split internally. FlashRuntime fans a grid out into one independent task per trial, runs them, and ranks the results, while your script stays plain sklearn with no FlashRuntime import. This tutorial builds a sweep from an ordinary script. You need pip install \"flashruntime[sklearn]\" (numpy + scikit-learn) — see Get started . 1. A plain sklearn script (flags in, metrics.json out) The only contract FlashRuntime asks: read hyperparameters from CLI flags, write a flat metrics.json . No FlashRuntime import anywhere. Copy import argparse import json def main(): parser = argparse.ArgumentParser() parser.add_argument(\"--model\", default=\"logreg\") parser.add_argument(\"--C\", type=float, default=1.0) parser.add_argument(\"--n_estimators\", type=int, default=50) args = parser.parse_args() from sklearn.datasets import make_classification from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import LogisticRegression from sklearn.model_selection import cross_val_score X, y = make_classification(n_samples=600, n_features=12, random_state=0) if args.model == \"logreg\": estimator = LogisticRegression(C=args.C, max_iter=500, random_state=0) elif args.model == \"rf\": estimator = RandomForestClassifier(n_estimators=args.n_estimators, random_state=0) else: raise SystemExit(f\"unknown model {args.model!r} (logreg|rf)\") scores = cross_val_score(estimator, X, y, cv=3) metrics = { \"model\": args.model, \"C\": args.C, \"n_estimators\": args.n_estimators, \"accuracy_mean\": round(float(scores.mean()), 4), } with open(\"metrics.json\", \"w\") as f: json.dump(metrics, f, indent=2) print(metrics) if __name__ == \"__main__\": main() Run one trial by hand to confirm it works: Copy python train.py --model rf --n_estimators 100 2. Fan a grid out with fr_sklearn.hpo(...) The integrations.sklearn adapter builds the workload from that script. Give it a grid; it expands the Cartesian product into one task per trial: Copy import flashruntime as flash from flashruntime.integrations import sklearn as fr_sklearn run = flash.submit(fr_sklearn.hpo( \"train.py\", {\"model\": [\"logreg\", \"rf\"], \"C\": [0.1, 1.0], \"n_estimators\": [50]}, source=\".\", )) print(f\"state={run.state.value} trials={len(run.trials)}\") print(\"best:\", run.best_trial()) {\"model\": [\"logreg\", \"rf\"], \"C\": [0.1, 1.0], \"n_estimators\": [50]} expands to 2 × 2 × 1 = 4 trials. Each {placeholder} in the built command is filled from the trial's params, so one task receives --model rf --C 1.0 --n_estimators 50 and so on. Every trial's metrics.json is collected and recorded on run.trials , with its params merged in. 3. Read the winner Because hpo (via sweep ) sets outputs.primary_metric=\"accuracy_mean\" , run.best_trial() needs no arguments — it returns the trial with the highest accuracy_mean : Copy best = run.best_trial() # ranks by accuracy_mean (maximize=True) worst = run.best_trial(maximize=False) # or flip it by_other = run.best_trial(metric=\"C\") # or rank by any reported key best_trial(metric=None, maximize=None) falls back to the OutputSpec defaults the adapter set; pass metric= / maximize= to override. It returns None if no trial reported the metric. How the fan-out stays correct Sequential and isolated. flash.submit() runs one trial at a time and copies each trial's metrics.json out before the next trial can overwrite it — so a trial's outputs are always its own. Independent checkpoint trees. Each trial gets its own job-scoped checkpoint tree, so trials never cross-contaminate. (This matters for checkpointed workloads; a pure sklearn fit has none.) Two API shapes. hpo(script, grid, **kwargs) is grid sugar over sweep(script, task_params, *, source=\".\", metric=\"accuracy_mean\", maximize=True, python=\"python\") . Use sweep directly to pass an explicit list of param dicts (e.g. a hand-picked, non-Cartesian set). Add a restart budget the same way as any run: flash.submit(..., max_restarts=1) retries a transient trial failure and fails fast on a deterministic one — see the fault-tolerance tutorial . Where to go next scikit-learn guide — the adapter reference and the \"distribute across runs, never inside .fit() \" rule. ConvNet tutorial — the PyTorch DDP + checkpoint-resume story. SDK reference — every Run attribute and submit() argument."}, {"url": "tutorials/fault-tolerance.html", "title": "Tutorial: automatic recovery, explained", "text": "Tutorial: automatic recovery, explained When a run fails, FlashRuntime does not guess what to do. It turns the failure into typed signals , classifies them into one failure class, and looks that class up in a versioned, deterministic policy table . Same failure + same policy version ⇒ same action, every time. No LLM, no scoring, no learned model in the loop. This tutorial follows one crash all the way through that pipeline, using the local flash.submit() path. It assumes you have done the ConvNet tutorial , whose kill-and-resume run is the worked example here. The budget: max_restarts Recovery on the local path is opt-in through one argument: Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit( fr_torch.ddp(\"train.py\", source=\".\", nproc_per_node=1, script_args=\"--steps 24 --checkpoint-every 8 --kill-at-step 8\"), output_dir=\"out/convnet\", max_restarts=1, # the automatic fault-tolerance budget (default 0 = no retry) ) max_restarts=0 (the default) means \"run once, no retry\" — the original behavior. Any higher number is the recovery budget: on a FAILED attempt, FlashRuntime consults the policy and, unless the failure is a deterministic application error, relaunches the same spec from the job-scoped checkpoint, up to that many times. Step 1 — signals: what a local process can actually tell you The first thing FlashRuntime does with a dead attempt is translate it into FailureSignals . On the local path the translator is recovery.signals.from_local_launch(exit_code, log_tail) , and it is deliberately narrow : a single local process can only evidence two things — its OS exit code and the tail of its captured stdout+stderr. It never fabricates node, accelerator, communication, or storage signals it cannot observe; those classes belong to the distributed coordinator, not a local launcher. It is a transparent lookup table, not an inference engine. The rules, checked top to bottom, first match wins: # Evidence Signals produced 1 exit 0 / never started neutral (nothing broke — a guard that keeps the function total) 2 a named deterministic exception on a traceback's terminal line ( ModuleNotFoundError: , NameError: , SyntaxError: , ImportError: , IndentationError: , AttributeError: ) exit_deterministic=True 3 a bare Traceback that is not the torchrun wrapper exit_deterministic=True 4 everything else (signal death / OOM / bare SystemExit / torchrun ChildFailedError ) plain crash (transient) Two subtleties worth knowing, because they are where naive implementations get it wrong: Rule 2 is anchored to the start of the traceback's terminal line, not a bare substring. CPython prints the failing exception type at the start of the final traceback line, while prose only ever mentions a name mid-line. The classic false positive is the startup log ... WARNING ImportError: flash_attn not available, falling back to eager — a transient run that a substring scan would wrongly fail fast. Line-anchoring ignores the name in prose and fires only on a real terminal. torchrun wraps every worker death — transient ones included — in a ChildFailedError and prints its own traceback. That would trip rule 3, so rule 3 is disqualified whenever ChildFailedError is present, letting the death fall through to the transient default (rule 4). This is exactly why the ConvNet kill-and-resume run recovers instead of failing fast. Step 2 — classify: signals to one failure class recovery.classify(signals) maps the signals to exactly one FailureClass , precedence-ordered so systemic evidence beats node evidence beats process evidence beats application evidence — a worker crash during a correlated incident is the incident, not the worker. The classes classify() can produce locally (the full FailureClass enum has two more — data_error , network_degradation — reachable only with coordinator-side signals): Copy correlated_incident systemic: many failures in the window — stop acting control_plane_failure coordinator unreachable preemption spot capacity reclaimed accelerator_failure GPU / driver / XID fault node_loss heartbeat lost / node unreachable communication_error NCCL / RCCL / rendezvous storage_timeout object-store errors artifact_corruption hash mismatch / failed validation application_error deterministic bug (exit_deterministic) ── fail fast worker_crash non-deterministic process death ── one fresh attempt unknown nothing matched On the local path only the last three are reachable, because those are the only classes a single process can evidence. A crashed torchrun worker with no deterministic terminal is a worker_crash . Step 3 — decide: class × mode to a typed action recovery.decide(failure_class, mode) is a pure table lookup returning a typed RecoveryDecision stamped with POLICY_VERSION . The mode matters because the same failure has a different blast radius: independent_tasks (a fan-out sweep) — one task retries; the others are untouched. coordinated_training (a torchrun group) — a lost rank stops the whole group, so recovery is a whole-group restart from the latest valid checkpoint (NCCL collective state is not repairable in place). flash.submit() picks the mode from the workload: a fan-out is independent_tasks , anything else (including a torchrun DDP run) is coordinated_training . A few rows of the table: Failure class independent_tasks coordinated_training application_error fail_job (retrying burns money on a bug) fail_job worker_crash retry_task restart_group (from latest valid checkpoint) node_loss retry_task (cordon + requeue) replace_node storage_timeout pause_job (don't burn compute on dead storage) pause_job correlated_incident freeze_automation (no retry storms) freeze_automation The most important action in the table is freeze_automation : during a correlated incident, the policy's job is to know when to stop acting. Putting it together: the ConvNet crash The kill-and-resume run from the ConvNet tutorial produces exactly this trail in run.events ( FAILURE_CLASSIFIED is amber and RECOVERY_ACTION_SELECTED is cyan on the live page): Copy LAUNCH_STARTED task-000 launched (pid ...) FAILURE_CLASSIFIED task-000: worker_crash (exit 1) RECOVERY_ACTION_SELECTED task-000: restart_group — a lost rank stops the group — restart all workers from latest valid checkpoint LAUNCH_STARTED task-000-r1 launched (pid ...) Reading it back through the pipeline: the torchrun child raised SystemExit(3) ; torchrun re-raised a ChildFailedError (so rules 2 and 3 did not fire) and exited non-zero → signals say \"plain crash\" → classify returns worker_crash → decide(worker_crash, \"coordinated_training\") returns restart_group . The retry ( task-000-r1 ) keeps the same job id, so its ft.prepare() finds the predecessor's newest valid manifest and resumes — resumed_from: 8 , not a restart from zero. And the fail-fast case: introduce a NameError in the script and the terminal traceback trips rule 2 → application_error → fail_job . FlashRuntime does not spend your restart budget re-hitting a deterministic bug; it fails immediately and tells you. Honest scope This is the local signal surface. from_local_launch sees an exit code and a log tail — nothing more. The richer classes ( node_loss , accelerator_failure , communication_error , correlated_incident ) are real and fully policy-covered, but they are evidenced by the distributed coordinator (leases, heartbeats, health signals), not by one local process. The taxonomy and policy table are the same in both places. Recovery never changes the math. A resumed run restores a verified, topology-compatible checkpoint and continues; determinism (fixed seeds, the seed-0 DistributedSampler ) is what makes a resumed run reproduce the uninterrupted one. Where to go next Architecture — recovery as one of the four axes, with the leases and manifests it relies on. SDK reference — submit(..., max_restarts=N) and the Run event log. JobSpec & isolation guide — how the same recovery machinery runs on a coordinator, under leases."}, {"url": "guides/pytorch.html", "title": "PyTorch guide", "text": "PyTorch guide FlashRuntime operates your PyTorch job — it never rewrites your model. You keep the framework, the model, the loop, and the loss you already have; FlashRuntime launches your command, injects the environment it promises, tracks metrics, validates checkpoints, retries on failure, and collects artifacts. This is ADR-0003's fourth axis in practice: recipes integrate user code . FlashRuntime plans, launches, observes, and recovers; the distributed math is always done by PyTorch ( torchrun , DDP). There are two paths, and both operate unmodified torch code. For a full worked walkthrough, do the ConvNet tutorial . Path 1 — a script that is already DDP-ready If your script already calls dist.init_process_group() and wraps its model in DistributedDataParallel itself, there are zero code changes . The adapter just builds the torchrun command: Copy import flashruntime as flash from flashruntime.integrations import pytorch as fr_torch run = flash.submit(fr_torch.ddp( \"train.py\", source=\"examples/user_pytorch_vanilla\", nproc_per_node=2, # 2 processes on this host — gloo/CPU works, no GPU script_args=\"--steps 100\", )) print(run.state.value, run.trials) ddp(script, *, source=\".\", nproc_per_node=2, nnodes=1, script_args=\"\", env=None) emits torchrun --nproc-per-node=N --nnodes=1 --standalone --local-addr=127.0.0.1 <script> <args> . --local-addr=127.0.0.1 pins the advertised rendezvous address to loopback (otherwise torchrun advertises socket.getfqdn() , which on some macOS DNS setups is unresolvable and the run hangs before spawning a process). nnodes > 1 raises NotImplementedError today — multi-node rendezvous is a launcher concern for a later slice. --standalone is single-node by definition. Path 2 — flashruntime.torch , the optional in-script helper For a script you are willing to touch, one import makes it both launch-anywhere and fault-tolerant, without rebuilding any framework machinery. The surface is three verbs plus read-only launch-fact accessors: Copy import flashruntime.torch as ft model, optimizer, loader = ft.prepare(model, optimizer, loader) start = ft.start_step() # 0 fresh, >0 after a resume ... ft.checkpoint(model, optimizer, step=step, every=100) ft.log_metrics({\"step\": step, \"loss\": float(loss)}) if ft.is_main(): # ft.rank(), ft.world_size() too ... prepare(model, optimizer=None, dataloader=None) — launched distributed ( WORLD_SIZE > 1 ) it initializes torch's own process group ( nccl on GPU, gloo on CPU), wraps the model in DistributedDataParallel , and swaps the DataLoader's sampler for a seed-0 DistributedSampler so each rank sees a disjoint, deterministically-shuffled shard. It then restores the newest valid checkpoint manifest if one exists, setting the resume step. Launched as plain python train.py , it is a no-op passthrough. CUDA device placement is wired. prepare selects nccl when CUDA is present, moves your model onto this rank's GPU before the DDP wrap, and binds DDP with device_ids / output_device — so a single-GPU box \"just works\" and you no longer call model.to(device) yourself. The everyday e2e tests exercise CPU / gloo ; real-GPU validation is tracked in the workspace progress log. Multi-node DDP ( nnodes > 1 ) is a later slice. One caveat: prepare rebuilds the DataLoader carrying over batch_size , collate_fn , num_workers , and drop_last — shuffle and pin_memory are not carried over (the DistributedSampler owns shuffling, at seed 0). checkpoint(model, optimizer=None, *, step, every=None) — rank 0 writes a checkpoint under the parts-first / manifest-last contract (the manifest is written last, so a half-written checkpoint is never latest_valid ). every=N no-ops except on multiples of N. Every rank synchronizes on a barrier so no one races past a partial write. log_metrics(dict) — rank 0 appends one JSON record per call to metrics.jsonl (streaming history; the live page's loss curve reads it). It never raises — metrics must never kill training. This is separate from the final metrics.json your script writes for run.trials . start_step() / rank() / world_size() / is_main() / device() / backend() — the small read-only helpers ( device() / backend() let a script report where it actually trained, e.g. into metrics.json — how the GPU e2e proves the CUDA/nccl path). Full signatures in the torch helper reference . The same file, three ways: Command What runs python train.py --steps 200 single process, prepare is a passthrough torchrun --nproc-per-node=2 --standalone train.py DDP by hand flash.submit(fr_torch.ddp(\"train.py\", ...)) operated by FlashRuntime Determinism / bit-exact resume Keep the script deterministic on CPU (fixed seeds; the seed-0 DistributedSampler repeats its order every epoch) and a killed-and-resumed run reproduces the uninterrupted result — recovery must not change the math. There is one alignment constraint: on resume the for loop restarts the dataloader at batch 0, so the resumed step must land on an epoch boundary — a multiple of batches-per-rank-per-epoch. Keep --checkpoint-every a multiple of that. One output_dir is one workload. Resume works by reusing the job-scoped checkpoint tree under output_dir . Point a different workload at an output_dir that already holds another workload's checkpoints and prepare() will happily restore those foreign weights — silent wrong results, not an error. Use a fresh output_dir per workload ; reusing one for the same workload is exactly how kill-and-resume is meant to work. (Fan-out sweeps are safe automatically: each trial gets its own checkpoint tree.) Guardrail (ADR-0003 — do not rebuild Accelerate) flashruntime.torch wraps torch's own DDP and stops. There are no FSDP policies, no autocast, no DeepSpeed config in this surface. Users who want those use the real framework features directly — the launcher still launches such a script correctly, because launching is orthogonal to the strategy your code chooses. Adding another framework The PyTorch adapter is ~50 lines: it builds a torchrun command string and returns a CommandWorkload . That is the whole extensibility pattern — a new framework adapter is a small function under flashruntime/integrations/ that returns a CommandWorkload , reusing the same launch/collect/recover machinery. integrations.huggingface is literally a thin wrapper over this ddp() — see the Hugging Face guide — and integrations.sklearn is the same shape for the fan-out case. No core change is needed to teach FlashRuntime a new framework; you describe what to run , and the four axes handle the rest."}, {"url": "guides/sklearn.html", "title": "scikit-learn guide", "text": "scikit-learn guide FlashRuntime operates your scikit-learn job — it never rewrites your estimator. You keep the model and the scoring; FlashRuntime fans a grid out into independent tasks, runs them, collects each metrics.json , and ranks the results. The rule that shapes this whole adapter: sklearn work is embarrassingly parallel across runs , never inside a single .fit() . FlashRuntime fans a grid into one independent task per trial — it never tries to split one .fit() call, which would change the math. For a worked walkthrough, do the sklearn sweeps tutorial . The contract: flags in, metrics.json out Your script needs zero FlashRuntime imports . It reads hyperparameters from CLI flags and writes a flat metrics.json to its working directory. That is the entire contract — the same one every framework uses. examples/user_sklearn/train.py is plain sklearn end to end. Fan a grid out The integrations.sklearn adapter builds the workload from that script: Copy import flashruntime as flash from flashruntime.integrations import sklearn as fr_sklearn run = flash.submit(fr_sklearn.hpo( \"train.py\", {\"model\": [\"logreg\", \"rf\"], \"C\": [0.1, 1.0], \"n_estimators\": [50]}, source=\"examples/user_sklearn\", )) print(f\"state={run.state.value} trials={len(run.trials)}\") print(\"best:\", run.best_trial()) # ranks by outputs.primary_metric hpo(script, grid, **kwargs) expands a Cartesian grid ( {\"model\": [\"logreg\", \"rf\"], \"C\": [0.1, 1]} → 4 trials) and delegates to sweep . sweep(script, task_params, *, source=\".\", metric=\"accuracy_mean\", maximize=True, python=\"python\") takes an explicit list of param dicts — use it when you want a hand-picked, non-Cartesian set. Each {placeholder} in the built command is filled from the trial's params, so train.py receives --model rf --C 1.0 and friends. Because sweep sets outputs.primary_metric=metric , run.best_trial() needs no arguments — it returns the trial with the highest accuracy_mean (or lowest, when maximize=False ). Why the fan-out is correct by construction Sequential and isolated. flash.submit() runs one trial at a time and copies each trial's metrics.json out before the next trial can overwrite it. Independent trees. Each trial gets its own job-scoped checkpoint tree, so trials never cross-contaminate. Add fault tolerance the usual way. flash.submit(..., max_restarts=1) retries a transient trial failure and fails fast on a deterministic one (a bad flag combination that raises the same error every time). See the fault-tolerance tutorial . Adding another framework The sklearn adapter is a ~40-line function that builds a CommandWorkload with task_params set for fan-out. A new framework adapter follows the same pattern: a small function under flashruntime/integrations/ that returns a CommandWorkload describing what to run , then reuses the same launch/collect/rank machinery. The PyTorch adapter is the coordinated-run counterpart, and Hugging Face is a thin wrapper over it — no core change is needed to teach FlashRuntime a new framework."}, {"url": "guides/huggingface.html", "title": "Hugging Face guide", "text": "Hugging Face guide FlashRuntime operates your Hugging Face job — it never rewrites your Trainer . HF Trainer already wraps DDP/FSDP internally when it is launched by torchrun , so launching an HF job is just the PyTorch path . What integrations.huggingface adds is the callback seam that commits Trainer checkpoints as verified manifests and relays Trainer metrics. transformers is imported only inside your training process — never in FlashRuntime's core. Launching trainer(script, *, source=\".\", nproc_per_node=1, script_args=\"\") is a thin wrapper over the PyTorch ddp() adapter, so everything in the PyTorch guide about launching and multi-process applies unchanged: Copy import flashruntime as flash from flashruntime.integrations import huggingface as fr_hf run = flash.submit(fr_hf.trainer( \"train_hf.py\", source=\"~/hf-project\", nproc_per_node=1, script_args=\"--model_name_or_path bert-base-uncased\", )) The callback seam Inside your training script, wire the callback and the resume in the usual HF way: Copy from flashruntime.integrations import huggingface as fr_hf trainer.add_callback(fr_hf.flashruntime_callback()) # on_save -> manifest, on_log -> metrics resume = fr_hf.latest_checkpoint(training_args.output_dir) # newest VALID checkpoint dir, or None trainer.train(resume_from_checkpoint=resume) flashruntime_callback() builds a TrainerCallback whose on_save writes a verified manifest for checkpoint-<step>/ (rank 0 only) and whose on_log relays metrics through flashruntime.torch.log_metrics . The transformers import is paid inside this factory, in your process. latest_checkpoint(output_dir) returns the storage prefix of the newest checkpoint dir with a valid manifest ( None means fresh start) — pass it straight to resume_from_checkpoint . So a Trainer run gets the same verified, parts-first / manifest-last checkpoint guarantee as a hand-written loop: a half-written checkpoint is never selected for resume, because the manifest is written last. Adding another framework integrations.huggingface is the smallest possible adapter: trainer(...) delegates straight to ddp(...) , and the only HF-specific code is the callback that maps on_save / on_log onto FlashRuntime's manifest and metric contracts. That is the extensibility pattern — teach FlashRuntime a new framework by writing a small adapter under flashruntime/integrations/ that (a) returns a CommandWorkload describing what to launch, and (b), if the framework has its own callback/hook system, maps those hooks onto write_manifest / log_metrics . No core change is needed. See the PyTorch adapter for the base case."}, {"url": "guides/jobspec-and-isolation.html", "title": "JobSpec & isolation guide", "text": "JobSpec & isolation guide flash.submit() runs your workload on the local machine. To hand a command workload to a FlashRuntime coordinator — so nodes pull and run it under leases, heartbeats, and recovery — you compile it to the versioned wire form (a JobSpec ) and POST it. This guide covers that compile step and the isolation tier that decides which machines a task is allowed to land on. For the local path, see the PyTorch and scikit-learn guides. Compile to a JobSpec Copy from flashruntime.workloads.command import to_jobspec from flashruntime.protocol.v1alpha1 import ImageSpec jobspec = to_jobspec( workload, # a CommandWorkload name=\"my-sweep\", image=ImageSpec(repository=\"myrepo/trainer\", tag=\"2026.07-a1b2c3\"), ) # POST jobspec.model_dump() to POST /v1alpha1/jobs # (or from the CLI: flashruntime submit-spec spec.json) to_jobspec(workload, name, image=None) produces a JobSpec{execution.backend: \"leases\", workload.type: \"command\"} . A pinned image is required — remote runs must be reproducible, and the schema already rejects the tag latest . On the coordinator the command recipe expands the job into one TaskSpec per task_params entry (or a single task), each carrying an argv payload, its env, its artifact:// inputs, and its isolation requirement. Isolation tiers (fail closed) Every command task carries an isolation tier from workload.isolation : Tier Where it runs Meaning standard (default) your own machines, RunPod, trusted pools ordinary placement — runs anywhere sandboxed community / untrusted machines may only be leased to a node advertising sandbox_capable is True The placement gate ( scheduler.IsolationAwarePlacement ) is fail-closed on the security-relevant field, per the schema-security rule: A node counts as capable only when sandbox_capable is True — a truthy stand-in (the string \"false\" , 1 , \"yes\" ) does not count. Any tier that is not None / \"\" / \"standard\" (including a mistyped \"Sandboxed\" ) is treated as requiring capability — no silent downgrade. A sandboxed task never falls back to an uncapable node unless the workload explicitly sets isolation.allowFallback = True . So a sandboxed task will sit unclaimed rather than land on a node that cannot isolate it. That is the intended behavior: unsafe placement fails closed. What runs where today Local SDK path — works now. flash.submit() runs sklearn sweeps, 2-process CPU DDP (via gloo ), and kill-and-resume from checkpoints on this machine. All three are proven by the example e2e tests. Service-side command jobs — expansion, placement, and execution all work. POSTing a to_jobspec() workload expands it into leased tasks, places them fail-closed by isolation tier, and — with a FlashNode agent running --runner argv — executes the argv payload inside a hardened, network-isolated container and commits the result. sandboxed tasks are only ever placed on a node that advertises both sandbox_capable and argv_capable ; see the repo's docs/guides/donate-a-machine.md for exactly what that container confines (and does not). Later slices. Multi-node DDP ( nnodes > 1 rendezvous — not available on volunteer nodes even later, since --network none rules out rendezvous), result verification for untrusted volunteer nodes, remote providers (RunPod) with source packaging ( git_revision ), and flash.run(StrategyPlan) wiring are open follow-ups. For how a leased task recovers when a node disappears, see the fault-tolerance tutorial and the architecture page — the same failure taxonomy and policy table drive both the local path and the coordinator. Adding another framework Isolation and JobSpec compilation are framework-neutral : to_jobspec serializes any CommandWorkload , whatever built it ( fr_torch.ddp , fr_sklearn.sweep , fr_hf.trainer , or one you hand-construct). So a new framework adapter (see the PyTorch adapter ) gets coordinator submission and isolation-aware placement for free — it only has to return a CommandWorkload . Built-in task modules Besides command workloads, the coordinator ships three allowlisted task modules under flashml_workloads/ — sklearn_trial (hyperparameter trials), kmeans_shard / kmeans_driver (sharded K-means), and sgd_trainer (checkpointable SGD with bit-identical resume). They are reference workloads for the lease protocol, not a required path: they predate command workloads and remain the workspace e2e's proof fixtures. Their contract is documented in each module's docstring and in the repo's AGENTS.md ."}, {"url": "guides/federated-averaging.html", "title": "Federated averaging", "text": "Federated averaging Federated averaging (FedAvg) is FlashRuntime's answer to a question the PyTorch guide and the JobSpec & isolation guide both run into: what do you do when the machines that want to help train a model cannot talk to each other ? Volunteer nodes run their task containers with --network none (see the repo's docs/guides/donate-a-machine.md ) — no LAN, no internet, no way for one container to find another. Coordinated multi-process training (DDP, FSDP) needs the opposite: every rank must rendezvous with every other rank over a process group before the first all_reduce . On a volunteer pool those two requirements are irreconcilable, so FlashRuntime does not attempt coordinated training there at all. FedAvg sidesteps the rendezvous problem by never requiring it. Instead of ranks synchronizing gradients mid-step, rounds synchronize whole models between steps: The driver broadcasts the current weights as a plain artifact:// blob. Each participating node downloads the weights, trains independently for a fixed number of local steps on its own data shard, and uploads a weight delta (not the new weights — see below). Once enough deltas have committed, the driver averages them, applies the result to the broadcast weights, and starts the next round. No node ever needs to see another node's IP address, let alone open a connection to it. Every cross-node interaction is a PUT / GET against the coordinator's artifact store, which is exactly the same shape of traffic a volunteer node already does to pull its task inputs and push its results. That is why this is a round loop implemented as a driver chaining ordinary lease jobs ( flashml_workloads/fedavg_driver.py , flashml_workloads/fedavg_worker.py ) — the same \"pipelines are jobs chained by a driver, not a new execution mode\" pattern as the sharded-k-means POC — rather than a new backend. Why a delta, not the new weights Each worker's task uploads delta.json (the change it made to the weights it started from) alongside metrics.json . The driver averages deltas , not raw weight snapshots, because a delta is a direction that stays meaningful even if the weights it was computed against are no longer the newest ones — the exact situation a straggling volunteer produces when it finally reports in after the round has moved on. Averaging final weights directly would require every worker to have started from the same snapshot; averaging deltas only requires knowing what each worker started from, which the driver already does. The quorum rule, and why late deltas are discarded kmeans_driver (the other job-chaining driver in this codebase) requires every dispatched shard to report before it aggregates. FedAvg deliberately does not: run_fedavg(..., min_participants=N) aggregates as soon as N of the round's shards have committed, not when all of them have. This is not a shortcut — it is the correct policy for volunteer compute. Machines that donate spare cycles are unequal and unreliable by construction: laptops close, Wi-Fi drops, a slow machine might still be on local step 3 when a fast one has already finished. Requiring all of them before a round can proceed would let a single closed laptop stall every other participant's contribution indefinitely. Quorum aggregation lets the round move on as soon as it has a statistically meaningful sample. The corollary is what makes quorum aggregation safe rather than merely convenient: once the driver has read the quorum's deltas and applied them, any delta that commits afterward for that round is discarded , never folded into a later round. run_fedavg freezes the participant set at the moment quorum is reached and never re-reads that job's artifacts again ( fedavg_driver.py , run_fedavg ). A late delta was computed against weights that no longer exist by the time it arrives — the model has already moved past them — and applying it on top of a newer round's weights would not be \"one more contribution,\" it would silently corrupt the average with a step that was never actually taken from the current state. Discarding is the honest behavior; a driver that tried to be more \"inclusive\" here would be quietly wrong instead. tests/test_fedavg_convergence.py::test_round_completes_on_quorum_when_a_node_never_reports pins exactly this: three shards are dispatched but the test's agent pool is capped to exactly two successful claims and then stops claiming, so the third shard is never bound to any node and sits PENDING for the life of the test. The round still aggregates on the two that committed — with an exact participants == 2 assertion — rather than hanging until the deadline waiting for the shard nobody was ever going to serve. (The cap on claims, not the node count, is what makes the third shard genuinely abandoned: either registered node can claim either shard, so without the cap both nodes could sequentially serve all three before the driver's poll notices quorum.) What counts as a participant, and what the driver refuses Everything a volunteer node produces — the delta, the sample count, the metrics file, the filenames — is attacker-controlled input. Result verification (catching a node that lies about a delta it honestly computed) is a later milestone, but input validation and containment are not deferred: A participant is an accepted commit, not an uploaded file. The driver counts only keys that exactly match the round's dispatched task set ( jobs/{job_id}/shard-{i:03d}/metrics.json for i < num_shards ), and cross-checks them against the tasks the coordinator reports COMPLETED ( GET /v1alpha1/jobs/{id}/tasks ). Both halves are load-bearing: the agent uploads a task's output tree recursively, so a nested out/a/metrics.json would otherwise mint a second participant from one lease; and uploads happen before the commit is offered, so an attempt the coordinator rejected (lost lease, sha256 mismatch) would otherwise still be averaged in. Sample counts must be positive. Validating only the total is not enough — (delta=-999, n=-999) plus (delta=1.0, n=1000) totals a healthy 1 sample but yields a weight of 999001.0 where the honest step is 1.0 . A sample-weighted mean is only a convex combination when every count is positive. NaN and Inf are rejected, not averaged. Python's json both emits and parses NaN / Infinity , and NaN is absorbing: one non-finite value turns every weight NaN, and every later round then trains from NaN while the run still reports success. This one needs no attacker — a learning rate that diverges on one shard does it. fedavg_weights fails closed on any non-finite value entering the reduce or leaving apply_delta / subtract , naming the parameter and index. lease_seconds is bounded ( modea.MAX_LEASE_SECONDS , one hour). A lease deadline is the only thing that returns an abandoned task to the queue, so 1e9 would pin a shard to a closed laptop for ~31 years and inf overflows timedelta inside the coordinator's claim path. Artifact PUT is now authenticated and lease-scoped when the coordinator sets FLASHML_NODE_TOKENS (the per-machine-token slice): a node token can only write under jobs/{job}/{task}/ for a task it currently holds a live lease on. The round-weights key ( jobs/{job_id}/round-{round:03d}/weights.json ) belongs to no task and no node's lease, so a plain node token cannot write it — the driver instead authenticates with an operator token ( FLASHML_OPERATOR_TOKENS ), which is attributable but deliberately not lease-scoped, exactly because drivers are legitimate writers outside any lease (see docs/guides/donate-a-machine.md ). Result verification is still a separate, unbuilt concern: this scoping stops an unrelated node from overwriting the round weights, not from a participant lying about the delta it honestly computed. The flashml.yaml shape A federated-averaging round is submitted as an ordinary lease-mode job: Copy apiVersion: flashml.dev/v1alpha1 kind: Job metadata: name: fedavg-r000 spec: execution: backend: leases image: repository: local/tier1 tag: dev workload: type: federated_averaging parameters: round: 0 num_shards: 2 local_steps: 20 lr: 0.1 batch_size: 16 seed: 0 in_dim: 8 hidden: 16 out_dim: 2 dataset_size: 256 # weights: artifact://jobs/<prev-job>/round-000/weights.json # (omitted on round 0 — each worker seeds its own model from `seed`) isolation.tier is left at its default, \"standard\" , deliberately: unlike the argv runner tier for arbitrary bring-your-code jobs, a federated_averaging task's payload is a fixed, trusted module execution ( flashml_workloads.fedavg_worker ), so it does not need the sandboxed argv path and its argv_capable gate. A node only needs module_capable (fail-open — absent counts as capable) to be eligible. run_fedavg builds this JobSpec once per round and submits it as a new job ( flashml_workloads/fedavg_driver.py:_round_body ) — the round number is the only thing that changes between the driver's own resume points. The image and isolation tier are run_fedavg parameters ( image= , isolation_tier= ); the defaults above are this repo's e2e fixture image, which only works because SubprocessRunner ignores image entirely — a docker-tier volunteer needs a real, pullable reference. What this proves — and what it does not tests/test_fedavg_convergence.py runs this loop against a real coordinator over real HTTP: real job expansion, real leases, real local artifact storage ( FLASHML_LOCAL_ARTIFACTS_DIR ), and real commit-time sha256 validation on every uploaded artifact. Two independent worker \"agents\" (a few lines of urllib , standing in for flashnode work — see the test file's docstring for why an in-repo test cannot import flashnode directly) pull leases, train, and commit without ever talking to each other. The measured per-round mean loss across four rounds with two participating nodes: Copy round 0 participants 2/2 mean_loss 0.5361 round 1 participants 2/2 mean_loss 0.3781 round 2 participants 2/2 mean_loss 0.2548 round 3 participants 2/2 mean_loss 0.1757 converged: 0.5361 -> 0.1757 over 4 rounds ( scripts/fedavg_local_demo.py reproduces this and exits non-zero if the final round's loss is not below the first — a demo that prints numbers nobody checks is not evidence.) Read that number correctly: this proves collaborative training, not faster training. Two nodes did not finish training in half the wall-clock time of one — they trained sequentially through four rounds, each doing its own local steps, and the loss came down because their independently computed updates were combined. Nothing here claims a throughput or speed-up result; DDP/FSDP make that claim, on a coordinated pool that can rendezvous, and that claim is out of scope for volunteer nodes entirely (the repo's docs/guides/donate-a-machine.md has the full list of things the volunteer pool does not attempt, including \"no coordinated multi-process training\"). What FedAvg proves is that machines which cannot see or trust each other — and in the volunteer case, cannot even reach each other over the network — can still jointly move one model's loss in the right direction, coordinated entirely through the coordinator's leases and artifact store."}, {"url": "concepts/architecture.html", "title": "Architecture", "text": "Architecture FlashRuntime plans, launches, observes, and recovers a distributed ML job. It never reimplements the distributed math — that always belongs to your framework (PyTorch DDP/FSDP, torchrun , Ray, Hugging Face). This page explains the shape of the system: the four axes it is built from, and the three mechanisms — leases, manifests, recovery — that make \"runs to verifiably completed on unreliable machines\" a real guarantee rather than a slogan. The design decisions here are recorded in ADR-0003 ( Reliability runtime first; planner as an explainable feasibility filter ). The four orthogonal axes The central idea: getting machines, starting processes, configuring execution, and integrating user code are four independent concerns. Keeping them orthogonal is what lets the same job run on your laptop, on RunPod, or on a community pool without rewriting anything. Copy ┌──────────────────────────────────────────────────────────────┐ │ your training job │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ ┌─────────┐ ┌──────────┐ ┌────────────┐ ┌──────────────┐ │providers│ │launchers │ │ strategies │ │ recipes │ │─────────│ │──────────│ │────────────│ │──────────────│ │ get │ │ start │ │ configure │ │ integrate │ │machines │ │processes │ │ execution │ │ user code │ └─────────┘ └──────────┘ └────────────┘ └──────────────┘ RunPod, torchrun, DDP / FSDP2 / PyTorch, sklearn, local, local proc, zero3 offload, Hugging Face K8s pools leased node single-GPU (this is the workload layer) The rule that keeps them honest: Hugging Face / PyTorch code lives in the recipe (workload) layer — it is never a backend. The planner emits a backend-neutral StrategyPlan and never imports framework code (no import transformers / torch.distributed / ray inside the planner). Launching is orthogonal to the strategy your code chooses, which is why FlashRuntime can launch an FSDP or DeepSpeed script correctly without knowing anything about FSDP or DeepSpeed. The integrations.* adapters you use from the SDK ( fr_torch.ddp , fr_sklearn.sweep , fr_hf.trainer ) are the recipe axis in practice: each returns a CommandWorkload describing what to run , and the other three axes handle the rest. Leases — the Mode A reliability core A lease is a time-bounded right to run one task. It is the layer no existing distributed-ML library provides, so FlashRuntime builds it first (Mode A) before coordinated training (Mode B). A node claims a task, sends heartbeats to keep the lease alive, and commits the result idempotently; if the heartbeats stop, the lease expires and the task requeues — automatic recovery with no central decision required. Copy task: PENDING │ claim (a node takes a time-bounded lease) ▼ LEASED ──heartbeat──► LEASED ──heartbeat──► LEASED │ │ │ no heartbeat within TTL │ commit (validated, idempotent) ▼ ▼ EXPIRED ──requeue──► PENDING COMPLETED Key properties: Status is derived from an append-only ledger of events , never a hand-mutated field. \"What is this job's state?\" is always answered by replaying events, so the answer is reproducible and auditable. Commit is idempotent and validated. A result is accepted only if it matches the task's expected commit key (a sha256), so a duplicate or corrupt commit cannot poison the job. Lease state is durable. In-flight leases survive a coordinator restart (a SQLite-backed store); agents re-register on their own. Manifests — checkpoint validity by construction A checkpoint is only useful if you can trust it after a crash. FlashRuntime makes validity structural with a parts-first / manifest-last commit: the checkpoint's parts are written first, then — only after their hashes verify — the manifest is written last. No manifest, no checkpoint. Copy write step-000123/ ├─ model.pt (part) ─┐ ├─ optimizer.pt (part) │ written FIRST └─ ... ─┘ │ hashes verified └─ manifest ──────────┘ written LAST ✔ now \"latest_valid\" crash between parts and manifest ⇒ no manifest ⇒ never selected So a half-written checkpoint — the exact thing a crash tends to produce — can never look valid. Recovery restores only a verified, topology-compatible manifest (the newest whose world size and framework match), which is why a resumed run continues correctly instead of loading garbage. ft.checkpoint(...) in the torch helper writes under this contract; the Hugging Face callback commits Trainer checkpoints the same way. Recovery — typed, deterministic, logged When something fails, recovery is a pure function of the evidence , not a judgment call. Raw signals are classified into one failure class, and the class (with the execution mode) is looked up in a versioned policy table that returns a typed action. There is no LLM, no scoring, no learned model — same failure + same policy version ⇒ same action, always. Copy failure evidence classify() decide(class, mode) ──────────────── ───────────────────── ────────────────────────── exit code, precedence-ordered: table lookup, versioned: log tail, ───► systemic > node > ───► worker_crash + coordinated heartbeat loss, process > app → restart_group (from ckpt) health signals → one FailureClass app_error → fail_job (fast) correlated → freeze_automation The design commitments: Deterministic application errors are never retried — fail fast and tell the user. Burning capacity re-hitting a bug is the most expensive kind of \"recovery\". Blast radius depends on mode. A worker_crash costs one task retry in independent_tasks mode but a whole-group restart in coordinated_training (NCCL collective state is not repairable in place). Correlated incidents freeze automation. The policy's most important action is knowing when to stop acting — retry storms during a systemic incident are how orchestrators destroy trust. Every decision is logged with its failure class and human-readable reason, and emitted as FAILURE_CLASSIFIED / RECOVERY_ACTION_SELECTED events the live page and ledger both show. The fault-tolerance tutorial walks one real crash through this pipeline end to end. How it fits together Copy plan ──► launch ──► observe ──► recover (StrategyPlan, (providers + (leases + (classify + decide, explained) launchers) heartbeats + typed actions from manifests) the policy table) The runtime is the spine; the planner is an explainable feasibility filter that sits in front of it ( flash.plan() ), and the runtime's ledger is the planner's dataset. Everything above is usable without the cloud — a self-hosted local coordinator is a first-class mode, not a demo shim. See also: the SDK reference for the entry points, and the JobSpec & isolation guide for the coordinator wire form."}, {"url": "reference/sdk.html", "title": "Reference: SDK (`flashruntime`)", "text": "Reference: SDK ( flashruntime ) The flashruntime top-level package. The core is pydantic-only — import flashruntime pulls in only pydantic ; the bring-your-own-code helpers ( submit , CommandWorkload , integrations ) resolve lazily so the planner stays a minimal import. Signatures below are exact. Each entry says, in one line, why it exists . Running a workload flash.submit Copy def submit(workload, output_dir=None, wait=True, max_restarts=0, watch=None) -> Run: ... The local entry point — compiles a CommandWorkload into a launch spec, runs it as a real subprocess (once per param set), collects artifacts, and returns a Run . output_dir — where run.json and artifacts land; a temp dir if None . Reusing the same dir reuses the job id, so a checkpointed script resumes. wait — True drives the launch loop inline and returns a finished Run ; False returns immediately and drives it on a daemon thread (watch it live). max_restarts — the automatic fault-tolerance budget (default 0 = no retry). A FAILED attempt is classified and run against the versioned recovery policy; a deterministic app error fails fast, anything else relaunches from the job-scoped checkpoint up to this many times. watch — open the live viewer and record its URL on run.viewer_url ; None (default) auto-decides (on at an interactive terminal, off in pipes/CI). Run The result handle, fully populated by the time a wait=True submit() returns. Attribute / method Meaning run.state LaunchState — SUCCEEDED if every task succeeded, else FAILED run.trials list of parsed metrics.json dicts (one per task; fan-out merges its params ) run.artifacts list of collected file Path s run.output_dir root the run wrote under run.viewer_url live-page URL if watch opened one, else None run.events snapshot copy of the append-only event log run.attempts snapshot copy of the per-launch attempt rows run.run_json_path path to the viewer_v1 run.json the Run mirrors itself to run.wait(timeout=None) block until terminal (event-based, no poll); returns the state run.logs(tail_lines=200) captured stdout+stderr (tail) run.best_trial(metric=None, maximize=None) best trial by metric (defaults from the workload's OutputSpec ); None if none reported it Describing a workload flash.CommandWorkload Copy class CommandWorkload(BaseModel): command: str | list[str] # shlex-split (NO shell) or an argv list source: Source = Source() # where the user's code lives image: ImageSpec | None = None # pinned image (required only for the service path) env: dict[str, str] = {} inputs: dict[str, str] = {} # each value must be an artifact:// URI outputs: OutputSpec = OutputSpec() resources: Requirements = Requirements() # resource hints (dormant locally; for future providers) isolation: IsolationSpec = IsolationSpec() mode: str = \"auto\" # auto | local | independent_tasks | coordinated checkpoint: CheckpointPolicy | None = None task_params: list[dict] | None = None # {name} placeholders filled per entry (Mode A fan-out) The user-facing description of a \"bring your own code\" workload — what to run and what FlashRuntime should do around it. It never describes how distributed math happens; that belongs to your code. command is shlex -split (there is no shell — pipes need an explicit command=\"bash -c '...'\" ). flash.Source Copy class Source(BaseModel): path: str = \".\" # local dir; ~ is expanded git_revision: str | None = None # reserved for remote packaging (later slice) Where the user's code lives — a flash.Source , not a bare string. flash.OutputSpec Copy class OutputSpec(BaseModel): prefix: str = \"artifact://jobs/{job_id}/\" collect: list[str] = [\"metrics.json\"] # globs resolved against the script's cwd primary_metric: str | None = None # the metrics.json key best_trial() ranks by maximize: bool = True What to keep after a run, and how to rank trials. metrics.json in collect (the default) is what populates run.trials . Planning a job flash.plan Copy def plan(request: PlanRequest) -> PlanReport: ... Deterministic, explainable strategy selection — turns model + hardware + objective into a ranked, explained PlanReport (no cluster required). The closed-form arithmetic is framework-import-free. flash.render Copy def render(report: PlanReport) -> str: ... Renders a PlanReport as human-readable text (the numbers, the chosen plan, and the rejected alternatives with their reasons). Copy import flashruntime as flash report = flash.plan(flash.PlanRequest( workload=flash.TransformerFineTune(model=\"Qwen/Qwen2.5-7B\", method=\"lora\"), resources=flash.Resources(gpus=4, gpu_type=\"RTX4090\"), objective=flash.Objective(mode=\"balanced\", deadline_minutes=240), )) print(flash.render(report)) Plan inputs Copy class Resources(BaseModel): gpus: int = 0 # 0 = CPU-only gpu_type: str | None = None # 'A100-40GB', 'L40S', 'RTX4090', ... vram_gb: float | None = None # per-GPU VRAM override hosts: int = 1 cpu_ram_gb: float = 32.0 cpu_cores: int = 8 hourly_cost_usd_per_gpu: float | None = None class Objective(BaseModel): mode: str = \"balanced\" # cheapest | fastest | balanced | reliable max_cost_usd: float | None = None deadline_minutes: float | None = None allow_quantization: bool = True allow_cpu_offload: bool = True allow_nvme_offload: bool = False Workload intents: TransformerFineTune , PyTorchTraining , ClassicalML , IndependentTasks . See the planner walkthrough (in the repo under docs/planner/ ) for the estimator arithmetic. flash.run — designed, not yet built Copy def run(plan, coordinator_url=None): ... # raises NotImplementedError The plan-to-execution bridge. It is designed (the docstring describes the intended pipeline) but not implemented, so it raises NotImplementedError rather than half-running. Today you flash.plan() and submit the JobSpec yourself. Related references Integrations — the fr_torch / fr_sklearn / fr_hf adapters that build a CommandWorkload for you. torch helper — the flashruntime.torch surface (three verbs + read-only accessors). CLI — the flashruntime command."}, {"url": "reference/integrations.html", "title": "Reference: integrations (`flashruntime.integrations`)", "text": "Reference: integrations ( flashruntime.integrations ) Framework adapters that build a CommandWorkload for you. Each is a small, framework-neutral function — the recipe axis of the four-axis architecture. No adapter imports its framework at the top level (the import is paid only in your training process). Signatures below are exact; each entry says, in one line, why it exists . integrations.pytorch ( fr_torch ) Copy def ddp( script, *, source=\".\", nproc_per_node=2, nnodes=1, script_args=\"\", env=None, ) -> CommandWorkload: ... Builds the torchrun launch command for a PyTorch script — the launch conventions only; DDP is wired by your code (or flashruntime.torch.prepare ). Emits torchrun --nproc-per-node=N --nnodes=1 --standalone --local-addr=127.0.0.1 <script> <args> . nnodes > 1 raises NotImplementedError (multi-node rendezvous is a later slice). See the PyTorch guide . integrations.sklearn ( fr_sklearn ) Copy def sweep( script, task_params, *, source=\".\", metric=\"accuracy_mean\", maximize=True, python=\"python\", ) -> CommandWorkload: ... def hpo(script, grid, **kwargs) -> CommandWorkload: ... sweep — one independent task per params dict; sets outputs.primary_metric=metric so run.best_trial() needs no arguments. hpo — Cartesian-grid sugar over sweep ( {\"model\": [\"logreg\", \"rf\"], \"C\": [0.1, 1]} → 4 trials). Distributes across runs, never inside a single .fit() . See the scikit-learn guide . integrations.huggingface ( fr_hf ) Copy def trainer(script, *, source=\".\", nproc_per_node=1, script_args=\"\") -> CommandWorkload: ... def latest_checkpoint(output_dir) -> str | None: ... def flashruntime_callback(): ... # returns a transformers TrainerCallback trainer — a thin wrapper over pytorch.ddp() ; launching an HF Trainer job is just the PyTorch path (Trainer wraps DDP/FSDP internally under torchrun ). latest_checkpoint — newest checkpoint-* dir with a valid manifest (or None ); pass straight to trainer.train(resume_from_checkpoint=...) . flashruntime_callback — a TrainerCallback whose on_save commits a verified manifest and whose on_log relays metrics through flashruntime.torch.log_metrics . The transformers import is paid inside this factory, in your process. See the Hugging Face guide . The extensibility pattern Every adapter returns a CommandWorkload and reuses the same launch/collect/recover machinery. To teach FlashRuntime a new framework, write a function under flashruntime/integrations/ that returns a CommandWorkload describing what to run (and, if the framework has hooks, maps them onto write_manifest / log_metrics ). No core change is required — that is the four-axis payoff. The SDK reference documents the CommandWorkload shape you build."}, {"url": "reference/torch-helper.html", "title": "Reference: torch helper (`flashruntime.torch`)", "text": "Reference: torch helper ( flashruntime.torch ) The optional in-training-script helper: one import ( import flashruntime.torch as ft ) makes a PyTorch script both launch-anywhere and fault-tolerant. torch is imported inside these functions only — FlashRuntime's core never depends on it. The surface is three verbs plus read-only launch-fact accessors — a deliberate guardrail (ADR-0003: do not rebuild Accelerate). The boundary is capability , not count: there are no FSDP policies, no autocast, no DeepSpeed config here, and there never will be. Signatures are exact; each says, in one line, why it exists . Copy def prepare(model, optimizer=None, dataloader=None): ... def checkpoint(model, optimizer=None, *, step, every=None) -> None: ... def log_metrics(metrics: dict) -> None: ... def start_step() -> int: ... def rank() -> int: ... def world_size() -> int: ... def is_main() -> bool: ... def device() -> str: ... # \"cpu\" or \"cuda:N\" — where prepare() put the model def backend() -> str | None: ... # \"gloo\"/\"nccl\", None when single-process prepare(model, optimizer=None, dataloader=None) Wires distributed execution and restores the newest valid checkpoint; returns the possibly-wrapped (model, optimizer, dataloader) triple. Launched distributed ( WORLD_SIZE > 1 ): initializes torch's own process group ( nccl on GPU, gloo on CPU), wraps the model in DistributedDataParallel , and swaps the DataLoader's sampler for a seed-0 DistributedSampler (each rank sees a disjoint, deterministic shard). On CUDA: selects nccl , moves the model onto this rank's GPU before the DDP wrap, and binds device_ids / output_device — you no longer call model.to(device) . Restores the newest valid checkpoint manifest if one exists, setting the resume step (read it with start_step() ). Launched as plain python train.py : a no-op passthrough. The rebuilt DataLoader carries over batch_size , collate_fn , num_workers , and drop_last . shuffle and pin_memory are not carried over — the seed-0 DistributedSampler owns shuffling. checkpoint(model, optimizer=None, *, step, every=None) rank 0 writes a resumable checkpoint under the parts-first / manifest-last contract (the manifest is written last, so a half-written checkpoint is never latest_valid ). every=N no-ops except on multiples of N. Every rank synchronizes on a barrier so no one races past a partial write. log_metrics(metrics: dict) rank 0 appends one JSON record per call to metrics.jsonl — the streaming history the live page draws as a loss curve. Never raises (metrics must never kill training). Separate from the final metrics.json your script writes for run.trials . start_step() -> int The first step the loop should run: 0 fresh, >0 after a resume (set by prepare() when it restores a checkpoint). Start your loop from it. rank() / world_size() / is_main() The small positional helpers, read from the launch environment: rank() is this process's RANK (default 0 ), world_size() is WORLD_SIZE (default 1 ), and is_main() is rank() == 0 — guard rank-0-only work (writing the final metrics.json , printing) with it. Usage shape Copy import flashruntime.torch as ft model, optimizer, loader = ft.prepare(model, optimizer, loader) start = ft.start_step() step = start while step < total_steps: for x, y in loader: ... step += 1 ft.checkpoint(model, optimizer, step=step, every=100) ft.log_metrics({\"step\": step, \"loss\": float(loss)}) if ft.is_main(): ... # write the final metrics.json The full worked example is the ConvNet tutorial ; the launch side is the PyTorch guide ."}, {"url": "reference/cli.html", "title": "Reference: CLI (`flashruntime`)", "text": "Reference: CLI ( flashruntime ) The flashruntime command ships with the [service] extra ( pip install \"flashruntime[service]\" ). It is the terminal front door to the same operations the SDK exposes: plan a job offline, run a command workload locally, and talk to a coordinator. The blocks below mirror the real --help output. Copy usage: flashruntime [-h] [--api API] {plan,submit,submit-spec,status,events,logs,cancel} ... positional arguments: {plan,submit,submit-spec,status,events,logs,cancel} plan evaluate a PlanRequest offline and print the strategy submit run a command workload locally (no API needed) submit-spec POST a JobSpec YAML to the coordinator — was `submit` before 0.1.0; renamed when `submit` became the local- workload front door options: -h, --help show this help message and exit --api API FlashRuntime API base URL --api names the coordinator base URL for the service-side subcommands ( submit-spec , status , events , logs , cancel ). plan — offline strategy selection Copy usage: flashruntime plan [-h] [--json] request_file positional arguments: request_file PlanRequest as .yaml or .json options: -h, --help show this help message and exit --json emit the full PlanReport as JSON Runs flash.plan() on a PlanRequest file and prints the explained strategy; --json emits the full PlanReport . No cluster required. submit — run a command workload locally Copy usage: flashruntime submit [-h] [--source SOURCE] [--task-params TASK_PARAMS] [--max-restarts MAX_RESTARTS] [--output-dir OUTPUT_DIR] [--watch | --no-watch] CMD positional arguments: CMD the command to run, e.g. 'python train.py --lr {lr}' options: -h, --help show this help message and exit --source SOURCE directory holding the user's code --task-params TASK_PARAMS JSON list of param dicts for Mode A fan-out --max-restarts MAX_RESTARTS automatic recovery budget --output-dir OUTPUT_DIR where run.json and artifacts land (default: temp dir) --watch, --no-watch open the live viewer (default: on at a terminal, off in pipes/CI) The terminal equivalent of flash.submit() . --task-params (a JSON list of param dicts) fills {name} placeholders in CMD for a fan-out sweep; --max-restarts is the automatic fault-tolerance budget. submit-spec — POST a JobSpec to the coordinator Copy usage: flashruntime submit-spec [-h] spec_file positional arguments: spec_file options: -h, --help show this help message and exit POSTs a JobSpec YAML/JSON file (e.g. one produced by workloads.command.to_jobspec ) to the coordinator named by --api . This was called submit before 0.1.0 — it was renamed when local submit became the default front door. status / events / logs / cancel — inspect a coordinator job Copy usage: flashruntime status [-h] job_id usage: flashruntime events [-h] job_id usage: flashruntime logs [-h] job_id usage: flashruntime cancel [-h] job_id Each takes a job_id and queries the coordinator at --api : status for the derived job state, events for the append-only ledger, logs for captured output, and cancel to stop a job. See the JobSpec & isolation guide for the wire form these service subcommands operate on, and the SDK reference for the in-process equivalents."}, {"url": "benchmarks.html", "title": "Benchmarks", "text": "Benchmarks Every number on this page is measured , never asserted. The tables below are rendered at docs-build time straight from the committed baseline JSON ( benchmarks/results/baseline-<host>.json ) — the docs cannot show a figure the suite did not produce. Each scenario states its hypothesis and its measurement method in its own source file ( benchmarks/scenarios/ ), so the methodology is auditable from the code alone, and every caveat and skip is printed verbatim in the notes under each table. Where a comparator (ray, accelerate) is not installed on the baseline machine, its row says so and its setup code is counted , not run, from the cited fixtures in benchmarks/scenarios/snippets/ — an honest line count, never a fabricated timing. Some figures here are small, zero, or negative: that is the suite working as intended. On a tiny CPU model, process-startup dominates wall-clock and a checkpoint write or a 40-step recompute falls below the run-to-run noise floor — the notes say so, and the size-independent guarantees (e.g. steps_not_recomputed ) are reported alongside. The value shows up at real model scale; the honesty shows up here. The suite is split into two sections, each rendered as its own table below. Performance measures the overheads a user pays for adopting flashruntime — launch overhead, per-checkpoint cost, submit latency, fan-out throughput, adoption line count — as wall-clock medians against a bare- torchrun or plain sequential baseline. Resilience measures the fault-tolerance guarantees themselves, under real failure injection rather than modelling: Correct classification — fault_recovery_matrix runs five distinct fault types (import error, mid-run SystemExit , a worker killed mid-run by an external SIGKILL , a SIGKILL inside the checkpoint-write window, and a corrupted newest part) and counts — from terminal run state, never asserts — how many the typed recovery taxonomy routes to the right action. Integrity under kill -9 — checkpoint_integrity fires repeated SIGKILL s that land inside the checkpoint write window; the parts-first / manifest-last commit means a torn write is never a valid checkpoint, so resume falls back to the last verified step. The naive torch.save comparator is killed in the same window and its corruption rate is reported beside ours. Goodput under a storm — crash_storm fans out a batch of trials with half of them armed to crash on their first attempt, then measures the fraction of useful work retained (and the tail actually recomputed) once every crashed trial auto-resumes from its checkpoint, with zero manual intervention. Measured MTTD/MTTR — lease_recovery_latency boots the real FastAPI coordinator over real loopback sockets, kills a worker mid-lease, and times detection (lease-expiry sweep) and recovery (re-claim → artifact commit) end to end, alongside steady-state claim/heartbeat round-trip latency. Each resilience number is COUNTED or timed from observable run state, never baked into an assertion. The failure injection lives in benchmarks/faults.py (crashy-trainer generation, kill -9 timing, part corruption), each scenario's measurement method is stated in its own source file under benchmarks/scenarios/ , and the long chaos loops carry a bench_stress marker so pytest -m bench_stress re-runs them. Reproduce the whole baseline yourself: Copy python -m benchmarks run --all --repeats 5 Run a single scenario, or a fast labelled smoke: Copy python -m benchmarks run --scenario recovery_economics python -m benchmarks run --all --smoke Measured on: os cpu cores ram_gb python torch flashruntime macOS-26.5.1-arm64-arm-64bit Apple M4 10 16.0 3.11.15 2.13.0 0.1.0 Reproduce every number below with: Copy python -m benchmarks run --all --repeats 5 Summary Performance scenario median unit p10 p90 repeats adoption_cost 7 lines to adopt 7 7 20 fanout_throughput 111 tasks/min 111 111 16 hpo_sweep 6.12 seconds 5.98 6.26 20 launch_overhead 0.0443 seconds -0.203 0.24 20 loop_overhead -1.25 ms/checkpoint -13.7 6.19 20 recovery_economics 0.259 seconds saved -0.368 2.44 20 submit_latency 0.0249 s (p50 cold) 0.0239 0.0259 20 Resilience scenario median unit p10 p90 repeats checkpoint_integrity 1 integrity_rate 1 1 20 crash_storm 16 completed/16 16 16 20 fault_recovery_matrix 5 correct/5 5 5 20 lease_recovery_latency 0.0035 s (MTTR) 0.0035 0.0035 20 adoption_cost Hypothesis: Adopting flashruntime is a handful of lines and a tiny dependency footprint — less code and a faster import than ray or accelerate. Measured: 7 lines to adopt (p10 7, p90 7, 20 repeats) comparator value accelerate_adopt_loc 8 flashruntime_core_deps 1 flashruntime_import_ms 74.9 torch_import_ms 692 adoption LOC = inserted/changed non-blank lines from a vanilla script to its framework-ready form (difflib); snippets are cited from each project's own docs LOC is deterministic — repeats do not vary it (import time is the median of 5 subprocess timings) ray not installed — import-time/dep-count comparator skipped accelerate not installed — import-time/dep-count comparator skipped fanout_throughput Hypothesis: A local flash.submit fan-out runs at a tasks/minute rate close to a plain sequential subprocess loop (fan-out is sequential by design), and its per-task overhead amortizes away as task duration grows. Measured: 111 tasks/min (p10 111, p90 111, 16 repeats) comparator value sequential_tasks_min 110 overhead_frac_05s 0.078 overhead_frac_2s 0.022 overhead_frac_5s 0.008 throughput = tasks / wall × 60, MEASURED over a single 16-task 0.5 s fan-out (p10==p90==median — one sweep, no repeat spread; the brief runs the full scenario once) overhead_frac = (flash_wall − ideal_serial_work) / flash_wall, where ideal_serial_work = count × duration = Σ INTENDED sleeps and NOTHING else — it ignores ALL process overhead (python startup, imports, spawn, collection), so the fraction deliberately CHARGES python startup to overhead. That is the honest 'ideal' floor; it never flatters flashruntime amortization curve counts: 0.5 s×16, 2 s×6, 5 s×4 — longer legs use fewer tasks (amortization is per-task, count-independent) to bound wall-clock. The 0.5 s fraction is LARGE (startup is a big slice of 0.5 s) and the 5 s fraction is small; that spread IS the finding, shipped as-is, never trimmed local fan-out is SEQUENTIAL by design (each trial's outputs are collected before the next runs), so this measures per-task ORCHESTRATION overhead and its amortization, NOT a parallel speedup — the flash and sequential rates are expected to be close hpo_sweep Hypothesis: flashruntime runs an 8-trial sweep with far less setup code than a hand-rolled loop or ray.tune, at comparable local wall-clock. Measured: 6.12 seconds (p10 5.98, p90 6.26, 20 repeats) comparator value sequential_s 5.98 peak_child_rss_mb 203 flash_setup_loc 10 sequential_setup_loc 16 ray_tune_setup_loc 22 flash.submit runs trials SEQUENTIALLY locally, so wall-clock ≈ the for-loop baseline; flashruntime's HPO value is orchestration + result collection + fault-tolerance, not local parallelism peak RSS is a shared RUSAGE_CHILDREN high-water mark across all trials (largest single child) ray.tune wall-clock is NOT measured here (ray not installed — a heavy, throwaway-venv comparator); its setup LOC is counted from the committed, cited snippet ray_tune_hpo.py launch_overhead Hypothesis: flash.submit adds under a second of wall-clock over a bare torchrun launch. Measured: 0.0443 seconds (p10 -0.203, p90 0.24, 20 repeats) comparator value flash_submit_s 2.53 bare_torchrun_s 2.46 overhead = flash.submit wall-clock minus identical bare torchrun, paired per repeat both launches share the same FLASHML_CKPT_DIR/OUTPUT_DIR isolation (a fresh temp) loop_overhead Hypothesis: The ft training loop adds negligible per-step cost; a checkpoint is a few to low-tens of ms. Measured: -1.25 ms/checkpoint (p10 -13.7, p90 6.19, 20 repeats) comparator value ft_steps_per_s 165 vanilla_steps_per_s 88.9 ft_vs_vanilla_wall_per_step_ratio 0.542 per-checkpoint = (ckpt=50 run - checkpoint-disabled run) / 4; startup+loop cancel in the delta a per-checkpoint figure at or below zero means the write cost is BELOW the run-to-run noise floor for this tiny model (a few-KB state dict) — checkpoints are effectively free here; a larger model would surface a positive cost steps/sec and the ratio are startup-dominated at this step count and compare DIFFERENT scripts (ft MLP vs vanilla Linear) — indicative, not identical-work checkpoint-every=0 raises ZeroDivisionError in the example; a sentinel > steps disables instead recovery_economics Hypothesis: Auto-resume from a checkpoint finishes a crashed run faster than a raw rerun-from-zero, and never recomputes the steps past the last checkpoint. Measured: 0.259 seconds saved (p10 -0.368, p90 2.44, 20 repeats) comparator value auto_resume_s 5 raw_rerun_from_zero_s 5.28 steps_not_recomputed 40 raw-torchrun cost modelled as t_crash + t_full (no checkpoint ⇒ rerun the whole job) auto-resume verified to restart from the step-40 checkpoint (trials[0].resumed_from == 40) seconds-saved is small here because torchrun startup (~2 s) dominates and the 40 recomputed steps of this tiny model cost only a fraction of a second; the saving scales with the compute between the last checkpoint and the crash — negligible at smoke size, hours on a real job steps_not_recomputed (40) is the size-INDEPENDENT guarantee: resume never re-does work past the last valid checkpoint, whatever a step costs submit_latency Hypothesis: flash.submit adds a small, roughly constant overhead before user code hits its first step — dominated by child-process launch, not flashruntime bookkeeping — invisible next to any real training run. Measured: 0.0249 s (p50 cold) (p10 0.0239, p90 0.0259, 20 repeats) comparator value cold_p95 0.0262 warm_p50 0.0258 warm_p95 0.035 phase_launch_s 0.0239 phase_child_s 0.0009 total = child.first_step_ts − parent.t0 (submit call → first training step), MEASURED from wall-clock instants the child stamps into metrics.json — never a baked figure phase_launch_s (submit call → child proc start) + phase_child_s (proc start → first step) == total by construction; interpreter startup lands in launch (it IS launch cost) cold = fresh output_dir per submit (cold dir/page cache); warm = 2nd..Nth of a reused dir (hot cache). The child is a fresh process either way, so cold≈warm — the small delta is the OS file-cache effect, MEASURED, not assumed away this is a CONSTANT per-launch overhead: invisible at real-training scale (minutes+); it matters only for very short jobs or very wide fan-outs — which is why we split and report it rather than amortize it away checkpoint_integrity Hypothesis: Under repeated kill -9s inside the checkpoint write window, flashruntime's parts-first/manifest-last commit means every resume lands on a hash-verified manifest (integrity_rate → 1.0), while a naive torch.save('latest.pt') overwriting one file in place is truncated by the same kill and fails to reload. Measured: 1 integrity_rate (p10 1, p90 1, 20 repeats) comparator value iterations 20 torn_writes_hit 20 window_missed 0 naive_torch_save_failure_rate 1 naive_torn_writes_hit 20 integrity is COUNTED from run state (terminal SUCCEEDED + a hash-verified latest manifest still present + a resume from a verified earlier step>0), never asserted — a mishandled in-window kill lowers the rate as a FINDING, not a test to fix integrity_rate = survived_hits / torn_writes_hit — the denominator is IN-WINDOW KILLS ONLY (20/20 iterations); window-missed iterations (0/20 — a clean uninterrupted run whose kill never landed) are EXCLUDED entirely, never counted as trivial 1.0 successes that would inflate the rate honesty: the write window is open a LARGE fraction of each step BY DESIGN (checkpoint_every=1, an 8 MB part per step widens the part-on-disk/manifest-absent gap), so hitting it is near-guaranteed — the claim is that flash SURVIVED every hit, NOT that hitting the window was hard honesty: torn_writes_hit certifies a kill during the manifest-absent COMMIT window (which includes the part-complete/manifest-pending sub-case), not necessarily a byte-torn part file — the guarantee under test is that a manifest-less step is never restored, however far its parts got mean resume step across torn-write hits: 1.0 (a verified EARLIER checkpoint, never the torn one) naive comparator: torch.save(state, 'latest.pt') overwritten in place each step, killed mid-write, then torch.load — raised on 20/20 iterations (mid-write kills landed on 20/20); observed failure modes, verbatim exception classes: EOFError×20 crash_storm Hypothesis: A 16-trial fan-out where every even trial crashes mid-run still completes 16/16 with zero human interventions — flashruntime auto-resumes each crash from its own checkpoint — at a bounded, MEASURED goodput and wall-clock cost. Measured: 16 completed/16 (p10 16, p90 16, 20 repeats) comparator value goodput_lower_bound 0.8 recompute_fraction 0 wallclock_penalty_fraction 0.243 manual_interventions 0 crashed_first_attempt 8 completions, goodput, crashed_first_attempt are COUNTED from run state (len(run.trials) and each trial's metrics steps/resumed_from), never asserted — a storm that completes <16/16 ships as the measured number, a FINDING goodput_lower_bound = Σ steps / Σ (steps + resumed_from): a WORST-CASE charge that re-counts each crashed trial's resumed_from AS IF it were recomputed (it was NOT — checkpointing preserved it), making the fraction a pessimistic lower bound that can never flatter flashruntime. The key NAMES the computation; it is not an executed-steps measurement recompute_fraction = Σ (crashed_at − resumed_from) / Σ steps, MEASURED from each crashed trial's ACTUAL crash step (faults.py writes crashed_at.json just before the raise; the resumed attempt folds it into metrics): the real redundant work, 0 when the crash fires AT a checkpoint step — the reuse-aware truth the lower bound deliberately over-charges manual_interventions = 0.0 is DERIVED: the max_restarts=1 recovery loop auto-resumed every WORKER_CRASH with no human in the loop (a bare torchrun needs one restart per crash) local fan-out is SEQUENTIAL by design (each trial's outputs are collected before the next runs), so both sweeps' wall-clock is a sum over trials — this measures fault-tolerant goodput, not throughput fault_recovery_matrix Hypothesis: Automated recovery does the right typed thing across a fault matrix — fail-fast on a deterministic bug, resume from the newest valid checkpoint on transient crashes/mid-run kills/mid-write kills/corruption — with zero human interventions where a bare torchrun needs one per fault. Measured: 5 correct/5 (p10 5, p90 5, 20 repeats) comparator value manual_interventions_flash 0 manual_interventions_torchrun_modelled 5 mean_recovery_s 0.256 mean_steps_preserved 3.3 torchrun_recovery_modelled_s 1.87 (a) import_error → FAIL_JOB fast-stop ✓ (1 attempt, 0 restarts burned) (b) systemexit_mid → auto-resume ✓ (resumed_from=4, 2 attempts) (c) mid-run external SIGKILL → resume ✓ (fired=True, resumed_from=2) (d) mid-write SIGKILL → resume from verified earlier step ✓ (fired=True, resumed_from=1) (e) corrupt newest part → resume from earlier valid ✓ (corrupt@8, resumed_from=6) correctness is COUNTED from run state (terminal state, attempts count, trials[0].resumed_from), never asserted manual_interventions_flash = cases the automation did NOT resolve correctly; torchrun modelled at 5 (no typed classification, no auto-resume — every fault needs a human to notice and resubmit) torchrun comparator (case b, bare torchrun argv, nproc=1): crash run exits nonzero with NO retry; recovery modelled as t_crash+t_full=1.87s (mean of 20; no checkpoint ⇒ rerun from zero) — labelled modelled, house convention lease_recovery_latency Hypothesis: Mode A detects a dead worker and requeues its task within lease_seconds + 2 s (the sweeper guarantee), and a healthy claim/heartbeat round-trip is a few ms — so recovery time (MTTR) is dominated by the tunable lease window, not coordinator overhead. Measured against the REAL coordinator over real sockets. Measured: 0.0035 s (MTTR) (p10 0.0035, p90 0.0035, 20 repeats) comparator value claim_rt_p50_ms 1.25 claim_rt_p95_ms 1.68 hb_rt_p50_ms 1.18 mttd_s 3.05 mttd_bound_s 5 roundtrips 20 measured against the REAL coordinator (flashruntime.service.app:app in a uvicorn SUBPROCESS on a free loopback port) over real HTTP — TestClient is in-process ASGI and would collapse the round-trip we measure, so it cannot back a latency claim first measured slice of the Stage-8 metrics debt (ledger-derived MTTD/MTTR): this exercises the lease loop end-to-end over the wire; the aggregate metrics reuse these events MTTD = B-reclaims-task − A-last-heartbeat, MEASURED; mttd_bound_s = lease_seconds (3.0) + sweeper period (2.0) = 5.0 s is the sweeper GUARANTEE — the observed value is faster because B's own claim sweeps first MTTD resolution is bounded by B's 50 ms claim-poll cadence (±one poll) MTTR = B-completes − B-claims, over the HONEST commit path: PUT the output bytes to /v1alpha1/artifacts/{commit_key} then complete with their real sha256 (server-side sha256-validated); it is ONE death cycle so p10/p90 == median — the N-wide spread is the claim round-trip in the comparators lease_seconds=3.0 chosen so the death cycle stays fast; the bound scales with whatever lease window a real deployment picks"}]