freesolo-flash-dev 0.2.31__tar.gz → 0.2.32__tar.gz

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: freesolo-flash-dev
-Version: 0.2.31
+Version: 0.2.32
 Summary: Flash — managed LoRA post-training (SFT/GRPO) for Freesolo environments, driven by the `flash` CLI
 Project-URL: Homepage, https://github.com/freesolo-co/flash
 Project-URL: Repository, https://github.com/freesolo-co/flash
@@ -51,15 +51,16 @@ Description-Content-Type: text/markdown
 
 # Flash
 
-Managed LoRA post-training service: SFT and GRPO on managed RunPod Flash GPUs.
-The allocator picks the cheapest validated RunPod GPU class that fits the run.
+Managed LoRA post-training service: SFT and GRPO on managed Flash GPUs.
+The allocator picks the cheapest validated managed GPU class that fits the run.
 
 ## Scope
 
 - `flash train <cfg.toml>` / control-plane `POST /runs` — submit a training job;
   one dedicated GPU per run, supervised server-side (stall watchdog, bounded
   auto-retry resuming from the last streamed checkpoint, endpoint GC).
-- `flash deploy`, `flash chat` — serving for trained adapters.
+- `flash checkpoints`, `flash deploy`, `flash chat` — serving for trained adapters,
+  including deployable intermediate RL checkpoints.
 - **Freesolo SDK environments.** Every run names a Freesolo environment id.
   Scaffold `environment.py` plus `datasets/train.jsonl`, upload `.` or another
   folder with `flash env push --name <name> <folder>`, then reference the
@@ -70,21 +71,22 @@ The allocator picks the cheapest validated RunPod GPU class that fits the run.
 ## Layout
 
 - `flash/catalog.py` — curated model catalog (Qwen3 dense supported tier;
-  Qwen3.5/3.6 experimental tier) + `model_policy = "allow"` VRAM-fit check + each
-  model's `thinking` capability (opt-in reasoning mode `thinking = true`)
+  Qwen3.5/3.6 experimental tier), VRAM-fit sizing, and each model's `thinking`
+  capability (opt-in reasoning mode `thinking = true`)
 - `flash/schema.py`, `flash/spec.py` — TOML → `JobSpec`
 - `flash/runner.py` — server-side run supervisor (durable job handle,
   retries, cost guard, endpoint GC)
-- `flash/providers/` — RunPod Flash provider code (pricing, gpus, durable
+- `flash/providers/` — managed GPU substrate code (pricing, GPU classes, durable
   submit/poll, preflight) behind the `base.Provider` protocol, with an
-  `allocator.py` that picks the cheapest fitting class
+  `allocator.py` that picks the cheapest fitting managed GPU class
 - `flash/engine/` — the on-GPU worker (TRL + colocated vLLM rollouts) and the
   shared recipe; SFT targets and RL rewards route through the active environment
   (task-specific grading lives with its example, not in the engine)
 - `flash/envs/` — environment machinery: registry and the adapter that loads
   Freesolo SDK environments onto the worker's interface
 - `flash env setup` — scaffold a starter local Freesolo env, `datasets/train.jsonl`,
-  and ready-to-run configs to start from
+  ready-to-run configs, and a `TRAINING.md` playbook with common failure modes and
+  mitigations
 - `flash/serve/`, `flash/server/` — adapter serving and the FastAPI control
   plane (run operator-side via the separate `flash-server` command)
 - `Dockerfile` — the control-plane image (used by the repo docker-compose)
@@ -101,12 +103,12 @@ uv run flash --help
 uv run flash-server                      # control plane (operator-side, run once)
 ```
 
-The control plane owns provider credentials: `RUNPOD_API_KEY` is always required,
-plus the shared `HF_TOKEN`.
+The control plane owns infrastructure credentials plus the shared `HF_TOKEN`.
 The artifact repo is platform-managed and environment-scoped (runs for the same environment share
 one private `Freesolo-Co/flashrun-<environment>-<hash>` repo, written by the operator `HF_TOKEN`);
-it is not a user knob and not an operator-wide env var. Clients authenticate with their freesolo
-API key (`flash login`).
+Flash uploads code under content-addressed prefixes and only reuses completed snapshots. The repo is
+not a user knob and not an operator-wide env var. Clients authenticate with their freesolo API key
+(`flash login`).
 
 ## Release channels
 
@@ -175,10 +177,10 @@ Use `choices[0].message.content` for the generated text. The run id is the adapt
 for serving. If the run is not deployed yet, `/v1/runs/<run_id>/chat` returns `409`
 with a hint to deploy first.
 
-Operators can also call the Modal serving app directly after the adapter is registered.
-The default serving app is `https://clado-ai--freesolo-lora-serving.modal.run`, and
-operators can point Flash at another serving app by setting `FREESOLO_SERVING_URL`.
-Use that same base URL when calling the app directly; pass the run id as `model`:
+Operators can also call the serving backend directly after the adapter is registered.
+The default serving backend is `https://clado-ai--freesolo-lora-serving.modal.run`, and
+operators can point Flash at another backend by setting `FREESOLO_SERVING_URL`.
+Use that same base URL when calling the backend directly; pass the run id as `model`:
 
 ```bash
 export FREESOLO_SERVING_URL=https://clado-ai--freesolo-lora-serving.modal.run

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/README.md

@@ -1,14 +1,15 @@
 # Flash
 
-Managed LoRA post-training service: SFT and GRPO on managed RunPod Flash GPUs.
-The allocator picks the cheapest validated RunPod GPU class that fits the run.
+Managed LoRA post-training service: SFT and GRPO on managed Flash GPUs.
+The allocator picks the cheapest validated managed GPU class that fits the run.
 
 ## Scope
 
 - `flash train <cfg.toml>` / control-plane `POST /runs` — submit a training job;
   one dedicated GPU per run, supervised server-side (stall watchdog, bounded
   auto-retry resuming from the last streamed checkpoint, endpoint GC).
-- `flash deploy`, `flash chat` — serving for trained adapters.
+- `flash checkpoints`, `flash deploy`, `flash chat` — serving for trained adapters,
+  including deployable intermediate RL checkpoints.
 - **Freesolo SDK environments.** Every run names a Freesolo environment id.
   Scaffold `environment.py` plus `datasets/train.jsonl`, upload `.` or another
   folder with `flash env push --name <name> <folder>`, then reference the
@@ -19,21 +20,22 @@ The allocator picks the cheapest validated RunPod GPU class that fits the run.
 ## Layout
 
 - `flash/catalog.py` — curated model catalog (Qwen3 dense supported tier;
-  Qwen3.5/3.6 experimental tier) + `model_policy = "allow"` VRAM-fit check + each
-  model's `thinking` capability (opt-in reasoning mode `thinking = true`)
+  Qwen3.5/3.6 experimental tier), VRAM-fit sizing, and each model's `thinking`
+  capability (opt-in reasoning mode `thinking = true`)
 - `flash/schema.py`, `flash/spec.py` — TOML → `JobSpec`
 - `flash/runner.py` — server-side run supervisor (durable job handle,
   retries, cost guard, endpoint GC)
-- `flash/providers/` — RunPod Flash provider code (pricing, gpus, durable
+- `flash/providers/` — managed GPU substrate code (pricing, GPU classes, durable
   submit/poll, preflight) behind the `base.Provider` protocol, with an
-  `allocator.py` that picks the cheapest fitting class
+  `allocator.py` that picks the cheapest fitting managed GPU class
 - `flash/engine/` — the on-GPU worker (TRL + colocated vLLM rollouts) and the
   shared recipe; SFT targets and RL rewards route through the active environment
   (task-specific grading lives with its example, not in the engine)
 - `flash/envs/` — environment machinery: registry and the adapter that loads
   Freesolo SDK environments onto the worker's interface
 - `flash env setup` — scaffold a starter local Freesolo env, `datasets/train.jsonl`,
-  and ready-to-run configs to start from
+  ready-to-run configs, and a `TRAINING.md` playbook with common failure modes and
+  mitigations
 - `flash/serve/`, `flash/server/` — adapter serving and the FastAPI control
   plane (run operator-side via the separate `flash-server` command)
 - `Dockerfile` — the control-plane image (used by the repo docker-compose)
@@ -50,12 +52,12 @@ uv run flash --help
 uv run flash-server                      # control plane (operator-side, run once)
 ```
 
-The control plane owns provider credentials: `RUNPOD_API_KEY` is always required,
-plus the shared `HF_TOKEN`.
+The control plane owns infrastructure credentials plus the shared `HF_TOKEN`.
 The artifact repo is platform-managed and environment-scoped (runs for the same environment share
 one private `Freesolo-Co/flashrun-<environment>-<hash>` repo, written by the operator `HF_TOKEN`);
-it is not a user knob and not an operator-wide env var. Clients authenticate with their freesolo
-API key (`flash login`).
+Flash uploads code under content-addressed prefixes and only reuses completed snapshots. The repo is
+not a user knob and not an operator-wide env var. Clients authenticate with their freesolo API key
+(`flash login`).
 
 ## Release channels
 
@@ -124,10 +126,10 @@ Use `choices[0].message.content` for the generated text. The run id is the adapt
 for serving. If the run is not deployed yet, `/v1/runs/<run_id>/chat` returns `409`
 with a hint to deploy first.
 
-Operators can also call the Modal serving app directly after the adapter is registered.
-The default serving app is `https://clado-ai--freesolo-lora-serving.modal.run`, and
-operators can point Flash at another serving app by setting `FREESOLO_SERVING_URL`.
-Use that same base URL when calling the app directly; pass the run id as `model`:
+Operators can also call the serving backend directly after the adapter is registered.
+The default serving backend is `https://clado-ai--freesolo-lora-serving.modal.run`, and
+operators can point Flash at another backend by setting `FREESOLO_SERVING_URL`.
+Use that same base URL when calling the backend directly; pass the run id as `model`:
 
 ```bash
 export FREESOLO_SERVING_URL=https://clado-ai--freesolo-lora-serving.modal.run

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/cli/__init__.py

@@ -66,7 +66,7 @@ _HELP_GROUPS: list[tuple[str, list[tuple[str, str]]]] = [
         "catalog",
         [
             ("models", "list supported base models"),
-            ("gpus", "list managed GPU classes with live $/hr"),
+            ("gpus", "list managed GPU classes with estimated $/hr"),
         ],
     ),
     (
@@ -226,7 +226,7 @@ def _build_parser() -> argparse.ArgumentParser:
     models = sub.add_parser("models", help="list supported base models")
     models.set_defaults(func=cmd_models)
 
-    gpus = sub.add_parser("gpus", help="list managed GPU classes with live $/hr")
+    gpus = sub.add_parser("gpus", help="list managed GPU classes with estimated $/hr")
     gpus.set_defaults(func=cmd_gpus)
 
     env = sub.add_parser("env", help="manage Freesolo environments")
@@ -268,7 +268,9 @@ def _build_parser() -> argparse.ArgumentParser:
     env_pull.set_defaults(func=cmd_env_pull)
 
     env_delete = env_sub.add_parser("delete", help="delete a published Freesolo environment")
-    env_delete.add_argument("env_id", help="the Freesolo environment id to delete, e.g. you/your-env")
+    env_delete.add_argument(
+        "env_id", help="the Freesolo environment id to delete, e.g. you/your-env"
+    )
     env_delete.add_argument(
         "-y",
         "--yes",

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/cli/commands.py

@@ -236,8 +236,8 @@ def cmd_env_setup(args) -> int:
             "[train]\n"
             "steps = 150\n"
             "lora_rank = 32\n"
-            "# GPU and the HF artifact repo are managed automatically by the platform: the GPU is\n"
-            "# the cheapest fitting class across providers, and each run gets its own artifact repo.\n"
+            "# GPU and HF artifacts are managed automatically by the platform: the GPU is\n"
+            "# the cheapest fitting managed class, and artifacts live in a private environment-scoped repo.\n"
         )
     sft = Path("configs/sft.toml")
     if not sft.exists():
@@ -248,8 +248,8 @@ def cmd_env_setup(args) -> int:
             "[train]\n"
             "epochs = 1\n"
             "lora_rank = 32\n"
-            "# GPU and the HF artifact repo are managed automatically by the platform: the GPU is\n"
-            "# the cheapest fitting class across providers, and each run gets its own artifact repo.\n"
+            "# GPU and HF artifacts are managed automatically by the platform: the GPU is\n"
+            "# the cheapest fitting managed class, and artifacts live in a private environment-scoped repo.\n"
         )
     training = Path("TRAINING.md")
     if not training.exists():
@@ -280,7 +280,7 @@ def cmd_models(args) -> int:
 
 
 def cmd_gpus(args) -> int:
-    """List RunPod GPU classes, VRAM, and $/hr."""
+    """List validated managed GPU classes, VRAM, and estimated $/hr."""
     from flash.providers.base import GPU_INFO
     from flash.providers.runpod.pricing import static_rates as runpod_static_rates
 
@@ -290,7 +290,7 @@ def cmd_gpus(args) -> int:
     )
     tip = (
         "Tip: GPU class selection is fully automatic — the submit-time allocator always picks the\n"
-        "cheapest validated RunPod class that fits the model, so you don't pin a GPU type."
+        "cheapest validated managed class that fits the model, so you don't pin a GPU type."
     )
     if render.styled():
         rows = [(info.name, info.vram_gb, runpod_rates.get(info.name)) for info in infos]
@@ -300,7 +300,7 @@ def cmd_gpus(args) -> int:
     def fmt_rate(v: float | None) -> str:
         return f"{v:>10.2f}" if v else f"{'-':>10}"
 
-    print(f"{'gpu':<16}{'vram':>6}{'runpod$/hr':>11}")
+    print(f"{'gpu':<16}{'vram':>6}{'$/hr':>11}")
     for info in infos:
         runpod_rate = runpod_rates.get(info.name)
         print(f"{info.name:<16}{info.vram_gb:>5}G{fmt_rate(runpod_rate):>11}")
@@ -483,8 +483,7 @@ def cmd_runs(args) -> int:
         spec = r.get("spec") or {}
         model = spec.get("model", "")
         algorithm = str(spec.get("algorithm") or "-").upper()
-        gpu, provider = render._run_where(spec, r.get("remote") or {})
-        where = f"{gpu}@{provider}" if provider else gpu
+        where = render._run_gpu(spec, r.get("remote") or {})
         print(
             f"{r['run_id']:<32}  {r['state']:<11}  {algorithm:<5}  "
             f"{r.get('cost_usd', 0.0):>8.4f}  {where:<22}  {model}"

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/cli/render.py

@@ -357,6 +357,21 @@ def _color_json(obj, depth: int) -> str:
     return _paint(json.dumps(obj), _GREEN)
 
 
+def _hide_provider_metadata(obj):
+    """Styled CLI details are for humans; keep backend provider names out of that view."""
+    if isinstance(obj, dict):
+        return {
+            k: _hide_provider_metadata(v)
+            for k, v in obj.items()
+            if k.lower() not in {"provider", "flash_arm"}
+        }
+    if isinstance(obj, list):
+        return [_hide_provider_metadata(v) for v in obj]
+    if isinstance(obj, str) and obj.lower() in {"runpod", "lambda"}:
+        return "managed"
+    return obj
+
+
 def version(value: str) -> str:
     """The wordmark + version."""
     mark = _paint(CLI_NAME, _ACCENT, "1")
@@ -388,13 +403,9 @@ def gpus_table(rows: list[tuple[str, int, float | None]], tip: str) -> str:
     return _safe(f"{header('gpus', 'managed GPU classes')}\n{table}\n\n{_dim(tip)}")
 
 
-def _run_where(spec: dict, remote: dict) -> tuple[str, str]:
-    """Return ``(gpu, provider)`` from remote (preferred) or spec fallback."""
-    provider = remote.get("provider") or (
-        "runpod" if remote else (spec.get("gpu") or {}).get("provider", "")
-    )
-    gpu = remote.get("gpu") or (spec.get("gpu") or {}).get("type", "")
-    return gpu, provider
+def _run_gpu(spec: dict, remote: dict) -> str:
+    """Human-facing GPU label. Provider metadata stays internal."""
+    return remote.get("gpu") or (spec.get("gpu") or {}).get("type", "")
 
 
 def runs_table(runs: list[dict]) -> str:
@@ -404,8 +415,7 @@ def runs_table(runs: list[dict]) -> str:
         spec = r.get("spec") or {}
         model = spec.get("model", "")
         algorithm = str(spec.get("algorithm") or "-").upper()
-        gpu, provider = _run_where(spec, r.get("remote") or {})
-        where = f"{gpu}@{provider}" if provider else gpu
+        where = _run_gpu(spec, r.get("remote") or {})
         color, uni, ascii_dot = _STATE_STYLE.get(str(r.get("state", "")).lower(), (_GRAY, "•", "-"))
         body.append(
             [
@@ -471,8 +481,7 @@ def _humanize_ts(value) -> str | None:
 def run_status(obj: dict) -> str:
     """A curated status panel for `flash status`, with the full JSON below for completeness."""
     spec = obj.get("spec") or {}
-    gpu, provider = _run_where(spec, obj.get("remote") or {})
-    where = f"{gpu} @ {provider}" if gpu and provider else (gpu or None)
+    where = _run_gpu(spec, obj.get("remote") or {}) or None
     pairs = [
         ("run id", _paint(obj.get("run_id", ""), _ACCENT2)),
         ("model", spec.get("model")),
@@ -492,7 +501,7 @@ def run_status(obj: dict) -> str:
     if obj.get("error"):
         pairs.append(("error", _paint(str(obj["error"]), _RED)))
     head = f"{header('status')}\n  {badge(obj.get('state', 'unknown'))}\n\n{_kv(pairs)}"
-    raw = f"{_dim('details')}\n{_json(obj)}"
+    raw = f"{_dim('details')}\n{_json(_hide_provider_metadata(obj))}"
     return _safe(f"{head}\n\n{raw}")
 
 
@@ -509,7 +518,7 @@ def object_panel(cmd: str, obj: dict, desc: str | None = None) -> str:
         if rid:
             line += "   " + _paint(rid, _ACCENT2)
         parts.append(line + "\n")
-    parts.append(_json(obj))
+    parts.append(_json(_hide_provider_metadata(obj)))
     return _safe("\n".join(parts))
 
 
@@ -586,7 +595,7 @@ def cost_panel(est) -> str:
         ),
         (
             "gpu",
-            f"{est.gpu} on {est.provider}  "
+            f"{est.gpu}  "
             f"{_dim(f'({est.gpu_vram_gb} GB; needs >= {est.required_vram_gb} GB)')}  "
             f"@ {money(est.gpu_hourly_usd, 2)}/hr",
         ),

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/cli/training_doc.py

@@ -20,10 +20,10 @@ actually produced, and decide — honestly — whether the model got better.
 
 Flash is a **managed** training service with a thin CLI/client. You author an
 environment (the task + its reward), publish it, and submit SFT or GRPO runs from a
-TOML config. Flash allocates the cheapest fitting GPU across providers, runs the job,
-streams logs back, and serves the result. You never handle provider credentials or a
-GPU — you authenticate once with a freesolo API key, and everything below is a `flash`
-CLI command.
+TOML config. Flash allocates the cheapest fitting managed GPU class, runs the job,
+streams logs back, and serves the result. You never handle infrastructure credentials —
+you authenticate once with a freesolo API key, and everything below is a `flash` CLI
+command.
 
 ### Install & authenticate
 
@@ -32,7 +32,7 @@ pip install freesolo-flash          # installs the `flash` CLI (import name is a
 flash login --api-key fslo_...       # or: export FREESOLO_API_KEY=fslo_...  (create a key at https://freesolo.co)
 flash whoami                         # confirm the identity behind your key
 flash models                         # supported base models (and which support `thinking`)
-flash gpus                           # managed GPU classes with live $/hr
+flash gpus                           # managed GPU classes with estimated $/hr
 ```
 
 ### The project layout (`flash env setup` created this)
@@ -105,12 +105,14 @@ id = "your-org/my-env"      # the id printed by `flash env push`
 steps = 150                 # GRPO is step-driven; SFT is epoch-driven (epochs = N)
 lora_rank = 32
 lora_alpha = 64
+# All GRPO/SFT knobs live under [train]. Do not add [grpo] or [sft] tables.
 ```
 
-GPU and the HF artifact repo are **fully managed** — there is no GPU knob; the allocator
-picks the cheapest class that fits, and each run gets its own artifact repo. Compose or
-tweak configs without editing files: `--config extra.toml` (deep-merge) and
-`--set key=value` (e.g. `--set train.steps=300`).
+GPU and HF artifacts are **fully managed** — do not pick `gpu.type` or set
+`train.hf_repo`; the allocator picks the cheapest validated managed GPU class that fits,
+and run artifacts are stored in a private environment-scoped repo with content-addressed
+Flash code snapshots. Compose or tweak configs without editing files: `--config
+extra.toml` (deep-merge) and `--set key=value` (e.g. `--set train.steps=300`).
 
 ### 4. Submit
 
@@ -139,6 +141,7 @@ flash deploy <run-id>            # serve the trained adapter (--step N for an in
 flash chat <run-id> -m "hello"   # chat with the deployed adapter
 flash deployments                # active serving endpoints
 flash undeploy <run-id>          # tear the endpoint down
+flash export --adapter-id <run-id> --repository <you>/<repo>  # copy adapter weights to your HF repo
 ```
 
 The rest of this file is about doing the above *well* — designing a reward that teaches,
@@ -185,6 +188,32 @@ If any box is unchecked, the run is not done improving — keep training, don't
 
 ---
 
+## Common Flash issues and mitigations
+
+Most bad Flash runs fail in a small number of predictable ways. Check these before
+spending another GPU run:
+
+| Issue | Symptom | Mitigation |
+| --- | --- | --- |
+| Environment id is blank or stale | `flash train --dry-run` fails, or the worker uses old reward/data | Run `flash env push --name my-env .` after every environment/data edit and paste the returned id into every config you submit. |
+| Local-only env path in config | Config validation says there is no local path mode | Publish first, then use the returned slug in `[environment] id`. `flash train` only runs published env ids, not local paths. |
+| Config knobs are in the wrong table | Validation rejects `[grpo]`, `[sft]`, or unknown `[train]` keys | Put `steps`, `epochs`, `group_size`, `max_tokens`, `temperature`, `max_length`, LoRA, and other training knobs under `[train]`. |
+| Trying to pin managed infrastructure | `gpu.type`, `train.hf_repo`, or `model_policy` changes do not do what you expected | Treat GPU choice, model policy, and the run artifact repo as managed. Tune the model, algorithm, environment, and `[train]` knobs instead. |
+| Secrets are not available on the worker | Reward code works locally but remote logs show missing API keys or auth failures | List secret names under `[environment] secrets = [...]`, export those env vars locally before submit, or put them in local `.env` / `.env.local`. Never put secret values in `[worker_env]` or hard-code them in the config. |
+| Wrong model / thinking setting | Config validation fails, or chat behavior does not match the run | Use `flash models`; set `thinking = true` only for supported models. Thinking is a training-time/run-level choice and serving preserves that parity, so `flash chat` does not expose an override flag. |
+| Thinking reward grades the wrong text | Rewards accidentally score hidden reasoning, or ignore reasoning you meant to inspect | By default, score the answer text. In thinking mode the response object is still string-compatible, but also exposes `.completion`, `.thinking`, and `.raw` when a reward intentionally needs those fields. |
+| All-zero or flat GRPO reward | `reward_mean` stays near 0 and outputs do not improve | Make the reward dense: give partial credit for parse/format/execution/correctness tiers, and log a separate clean `success` metric. Do not keep rerunning an all-zero reward. |
+| Reward rises but behavior is worse | Short, templated, malformed, or reward-hacked outputs score well | Deploy the adapter and probe real examples. Add hard validity gates before judge calls, penalize degenerate shortcuts, and judge the outcome rather than the surface string. |
+| Output is truncated | Correct-looking answers cut off mid-response or JSON is incomplete | Increase `max_tokens` for GRPO rollouts or `max_length` for SFT only after seeing truncation. Oversizing them by default just burns memory/cost. |
+| Infrastructure, CUDA, OOM, vLLM, or kernel failure | Run errors before useful metrics, often during setup/model load | Treat this as infrastructure pressure, not proof the model is too large. Read `flash status <run-id> --logs`, reduce footprint (`max_length`, `max_tokens`, `group_size`) if needed, and let Flash retry/allocate another fitting GPU class. |
+| Run looks stuck after disconnecting | Terminal stopped streaming but the job may still be alive | Ctrl-C detaches. Use `flash status <run-id> --follow` to reattach, `--logs` for the console/error tail, or `flash cancel <run-id>` if you intentionally want to stop it. |
+| Final checkpoint regresses | Last step is worse than an earlier checkpoint | Run `flash checkpoints <run-id>`, deploy a specific step with `flash deploy <run-id> --step N`, and compare with held-out probes before exporting or relying on the final adapter. |
+| Export fails before upload | CLI says no HuggingFace token | Pass `flash export --api-key hf_...`, or set `HF_TOKEN` in your shell, `.env`, or `.env.local`. Exports are private unless you pass `--public`. |
+| SFT loss improves but quality does not | Train loss falls while held-out behavior stalls or degrades | Keep a held-out split outside training. Deploy and score that split; if quality drops, reduce epochs or improve data instead of adding more passes. |
+| Cost surprises | A quick experiment uses more GPU time than intended | Start with `--dry-run` and `--cost`, cap steps/epochs for smoke tests, and scale only after reward/data wiring is proven. Setup time is reported for observability; customer cost is based on training-loop GPU time. |
+
+---
+
 ## Judge the run, don't just finish it
 
 - **Judge the trend, not a single number.** The proof of training is the curve:
@@ -259,6 +288,12 @@ attach is logged by name in the per-scorer breakdown — that is how the clean s
 rate becomes visible. Use the shaped `score` to confirm the model is learning *at all*,
 and judge the run on the explicit `success` metric.
 
+When `thinking = true`, score the final answer unless you intentionally need the
+reasoning trace. Flash passes a string-compatible response object to `score_response`;
+`str(response_text)` is the answer text, while `response_text.completion`,
+`response_text.thinking`, and `response_text.raw` are available for rewards that
+explicitly inspect the separated completion, reasoning, or original raw model output.
+
 ### Reward rules that prevent silent failure
 
 - **Return `0.0` explicitly — never let scoring raise.** An uncaught exception in
@@ -418,7 +453,7 @@ on a beyond-noise improvement.
 
 ## Treat crashes as infra, not model size
 
-> A CUDA / OOM / vLLM / kernel / provider error is an **infrastructure** problem, not a
+> A CUDA / OOM / vLLM / kernel / infrastructure error is an **infrastructure** problem, not a
 > sign the model is too big. Lower `max_length`, `max_tokens`, or `group_size` to shrink
 > the run's footprint and let the allocator retry onto the next fitting GPU class — do
 > **not** switch to a smaller model to make a crash disappear. That silently destroys
@@ -431,6 +466,8 @@ on a beyond-noise improvement.
 ```bash
 flash env setup                       # scaffold environment.py, datasets/, configs/, this file
 flash env push --name my-env .        # publish the environment; paste the returned id into [environment]
+flash env pull your-org/my-env        # download a published environment into the current folder
+flash env delete your-org/my-env -y   # delete a published environment
 flash train configs/rl.toml --dry-run # validate the config locally (no GPU, no charge)
 flash train configs/rl.toml --cost    # pre-flight USD estimate, then exit
 flash train configs/rl.toml           # submit and follow logs (Ctrl-C detaches; --background to skip following)
@@ -438,7 +475,15 @@ flash status <run-id>                 # state + accrued cost
 flash status <run-id> --logs          # reward/loss trend + worker console/error logs
 flash status <run-id> --follow        # stream a live run to completion
 flash runs                            # list your runs and their state/cost
+flash cancel <run-id>                 # stop a live run
+flash checkpoints <run-id>            # list deployable RL checkpoints
 flash deploy <run-id>                 # serve the trained adapter
+flash deploy <run-id> --step N        # serve a specific RL checkpoint
+flash chat <run-id> -m "probe"        # stream a reply from the deployed adapter
+flash deployments                     # list active serving deployments
+flash undeploy <run-id>               # tear down an active deployment
+flash export --adapter-id <run-id> --repository <you>/<repo>  # export final adapter
+flash export --adapter-id <run-id> --repository <you>/<repo> --step N  # export a checkpoint
 ```
 
 See the full reference at https://freesolo.co/docs.

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/cost/types.py

@@ -41,7 +41,9 @@ class RunConfig:
         # substrate up front (else it filters out every candidate -> confusing "no GPU fits").
         prov = (self.provider or "auto").strip().lower() or "auto"
         if prov not in ("auto", *PROVIDER_NAMES):
-            raise ValueError(f"unknown provider {self.provider!r} (auto, {', '.join(PROVIDER_NAMES)})")
+            raise ValueError(
+                f"unknown provider {self.provider!r} (auto, {', '.join(PROVIDER_NAMES)})"
+            )
         object.__setattr__(self, "provider", prov)
         if self.steps < 1:
             raise ValueError(f"steps must be >= 1, got {self.steps}")
@@ -86,7 +88,14 @@ class RunConfig:
             comp = None
             batch = self.batch_size if self.batch_size is not None else RECIPE.sft.effective_batch
             group = None
-        return replace(self, seq_len=seq, completion_len=comp, batch_size=batch, group_size=group, lora_rank=lora)
+        return replace(
+            self,
+            seq_len=seq,
+            completion_len=comp,
+            batch_size=batch,
+            group_size=group,
+            lora_rank=lora,
+        )
 
     def train_knobs(self) -> dict[str, int]:
         """The knob dict ``model_required_vram_gb`` consumes. Only an EXPLICIT batch_size is
@@ -141,7 +150,7 @@ class CostEstimate:
         """Multi-line itemized breakdown for CLI output."""
         lines = [
             f"Run        : {self.model_id}  [{self.method.upper()}, {self.steps} steps]",
-            f"GPU        : {self.gpu} on {self.provider} "
+            f"GPU        : {self.gpu} "
             f"({self.gpu_vram_gb} GB; run needs >= {self.required_vram_gb} GB) "
             f"@ ${self.gpu_hourly_usd:.2f}/hr",
             f"Setup      : {self.setup_seconds / 60:.1f} min (cold start: boot + deps + model load"

{freesolo_flash_dev-0.2.31 → freesolo_flash_dev-0.2.32}/flash/envs/adapter.py

@@ -410,6 +410,45 @@ def _github_response_message(payload: object) -> str:
     return ""
 
 
+def _github_tree_entries(ref: GitHubEnvironmentRef, treeish: str, context: str) -> list[dict]:
+    payload = _download_github_json(ref, _github_tree_url(ref, treeish), context)
+    if not isinstance(payload, dict) or not isinstance(payload.get("tree"), list):
+        raise RuntimeError(
+            f"GitHub path {context!r} is not an environment directory"
+            f"{_github_response_message(payload)}"
+        )
+    if payload.get("truncated"):
+        raise RuntimeError(
+            f"GitHub tree response for environment directory {context!r} was truncated"
+        )
+    entries = payload["tree"]
+    if not all(isinstance(entry, dict) for entry in entries):
+        raise RuntimeError("GitHub tree response included an invalid entry")
+    return entries
+
+
+def _resolve_github_directory_tree_sha(ref: GitHubEnvironmentRef, repo_dir: str) -> str:
+    treeish = ref.ref
+    current = ""
+    for part in [part for part in repo_dir.split("/") if part]:
+        entries = _github_tree_entries(ref, treeish, current or ref.ref)
+        match = next(
+            (
+                entry
+                for entry in entries
+                if entry.get("path") == part
+                and entry.get("type") == "tree"
+                and isinstance(entry.get("sha"), str)
+            ),
+            None,
+        )
+        current = f"{current}/{part}" if current else part
+        if match is None:
+            raise RuntimeError(f"GitHub path {repo_dir!r} is not an environment directory")
+        treeish = match["sha"]
+    return treeish
+
+
 def _download_github_directory(ref: GitHubEnvironmentRef, repo_dir: str, dest: Path) -> Path:
     """Download one GitHub directory into a repo-shaped tree under ``dest``."""
     repo_root = dest / "repo"
@@ -464,9 +503,10 @@ def _download_github_directory(ref: GitHubEnvironmentRef, repo_dir: str, dest: P
         (repo_root / path).mkdir(parents=True, exist_ok=True)
 
     create_dir(repo_dir)
+    package_tree_sha = _resolve_github_directory_tree_sha(ref, repo_dir)
     payload = _download_github_json(
         ref,
-        _github_tree_url(ref, f"{ref.ref}:{repo_dir}", recursive=True),
+        _github_tree_url(ref, package_tree_sha, recursive=True),
         repo_dir,
     )
     if not isinstance(payload, dict) or not isinstance(payload.get("tree"), list):