freesolo-flash-dev 1.4.5__tar.gz → 1.4.7__tar.gz

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/CHANGELOG.md

@@ -35,6 +35,13 @@ This file starts at 1.1.35. Earlier releases are not reconstructed here; use
   run recovery, deployment recovery and the background loops; a plane started beside it serves
   HTTP and leaves new runs queued for the holder until the old plane exits.
 
+- SFT rows accept `tools`: OpenAI-style function declarations, validated and normalized exactly
+  as a chat request's `tools`. They are rendered through the chat template into every training
+  view of the row (transcript, prompt boundary and loss-mask probe), so a tool-calling adapter
+  trains on the prompt it is served with. Previously every SFT row rendered without tools, so for
+  Qwen3.5 the `# Tools` block that serving prepends to the system turn never appeared in
+  training. The block is prompt only, so supervised tokens are unchanged. Rows without `tools`
+  render as before; `tools` on an image-bearing row is refused.
 - Custom training rows accept a rectangular `target_tokens` matrix, `[len(tokens)-1, K]`
   with K up to 64, scoring K tokens at every position instead of the next token. `weights`
   then has the same shape and logprobs come back with it, so `forward_backward` computes
@@ -155,6 +162,12 @@ This file starts at 1.1.35. Earlier releases are not reconstructed here; use
 
 ### Fixed
 
+- API-key verification bridges brief Freesolo identity-service outages. A key that verified
+  successfully within the last 5 minutes is accepted when verification is unavailable (transport
+  errors or 5xx), instead of failing the request with `503 Authentication service unavailable`.
+  Live verification still runs on every request while the service answers, and a definitive
+  rejection forgets the key, so revocation stays immediate. Long multi-turn rollouts previously
+  lost about 13% of episodes to these 503s.
 - Image SFT no longer stores processed pixel tensors in the training parquet. The worker keeps
   each row's small processor fields (image grid, token types) and its saved image files, and the
   training dataset rebuilds `pixel_values` per item, checking the stored grid. Before this,

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: freesolo-flash-dev
-Version: 1.4.5
+Version: 1.4.7
 Summary: Flash — managed LoRA post-training (SFT/GRPO) for Freesolo environments, driven by the `flash` CLI
 Project-URL: Homepage, https://github.com/freesolo-co/flash
 Project-URL: Repository, https://github.com/freesolo-co/flash
@@ -341,6 +341,13 @@ endpoint returns `409` with a hint to deploy first. Prefer this control-plane en
 directly: it enforces run ownership and forwards per-run serving options such as
 thinking-mode parity.
 
+Tool parameters use a restricted JSON Schema profile. Inside JSON arrays and objects,
+scalar types can be combined explicitly, for example `"type": ["string", "number",
+"boolean", "null"]` for spreadsheet cells. Empty schemas (`{}`), container unions,
+and unions on top-level function arguments are unsupported. Top-level arguments use
+native parameter text, where a string such as `"12"` and a number would be ambiguous;
+nested JSON preserves those types during generation, streaming, and history replay.
+
 ### Workload estimates (SFT)
 
 `flash train`, `--dry-run`, and `--cost` read the pinned environment's packaged dataset
@@ -352,8 +359,14 @@ The estimate tokenizes raw `input`/`output` fields plus the statically readable
 contract. SFT rows can instead supply `prompt_messages`, a nonempty chat-message list
 that is authoritative for both profiling and training. This preserves stored multi-turn
 histories and screenshot blocks; image-bearing transcripts use unpacked execution.
-The row still requires `input` for dataset identity. Environment-added prompts, few-shot
-examples, tool schemas, filters, and transformations are **not** executed, so real training
+The row still requires `input` for dataset identity. A row for a tool-calling agent can also
+supply `tools`, the same OpenAI-style function declarations a chat request sends. Flash
+validates them as serving does and renders them into the row's prompt through the chat template,
+so the adapter trains on the prompt it is served with: Qwen3.5, for example, prepends a
+`# Tools` block to the system turn. The block is prompt, not target, so it lengthens rows
+without adding supervised tokens. Without `tools`, a row trained for a tool-calling deployment
+never sees that block. Environment-added prompts, few-shot
+examples, environment-built tool schemas, filters, and transformations are **not** executed, so real training
 can retain fewer rows, truncate more often, and cost more than the estimate. If no readable dataset exists, cost, dry-run, and
 submit all fail before GPU allocation rather than billing a profiling run.
 

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/README.md

@@ -241,6 +241,13 @@ endpoint returns `409` with a hint to deploy first. Prefer this control-plane en
 directly: it enforces run ownership and forwards per-run serving options such as
 thinking-mode parity.
 
+Tool parameters use a restricted JSON Schema profile. Inside JSON arrays and objects,
+scalar types can be combined explicitly, for example `"type": ["string", "number",
+"boolean", "null"]` for spreadsheet cells. Empty schemas (`{}`), container unions,
+and unions on top-level function arguments are unsupported. Top-level arguments use
+native parameter text, where a string such as `"12"` and a number would be ambiguous;
+nested JSON preserves those types during generation, streaming, and history replay.
+
 ### Workload estimates (SFT)
 
 `flash train`, `--dry-run`, and `--cost` read the pinned environment's packaged dataset
@@ -252,8 +259,14 @@ The estimate tokenizes raw `input`/`output` fields plus the statically readable
 contract. SFT rows can instead supply `prompt_messages`, a nonempty chat-message list
 that is authoritative for both profiling and training. This preserves stored multi-turn
 histories and screenshot blocks; image-bearing transcripts use unpacked execution.
-The row still requires `input` for dataset identity. Environment-added prompts, few-shot
-examples, tool schemas, filters, and transformations are **not** executed, so real training
+The row still requires `input` for dataset identity. A row for a tool-calling agent can also
+supply `tools`, the same OpenAI-style function declarations a chat request sends. Flash
+validates them as serving does and renders them into the row's prompt through the chat template,
+so the adapter trains on the prompt it is served with: Qwen3.5, for example, prepends a
+`# Tools` block to the system turn. The block is prompt, not target, so it lengthens rows
+without adding supervised tokens. Without `tools`, a row trained for a tool-calling deployment
+never sees that block. Environment-added prompts, few-shot
+examples, environment-built tool schemas, filters, and transformations are **not** executed, so real training
 can retain fewer rows, truncate more often, and cost more than the estimate. If no readable dataset exists, cost, dry-run, and
 submit all fail before GPU allocation rather than billing a profiling run.
 

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/cli/scaffold/TRAINING.md

@@ -461,13 +461,10 @@ described above.
 Never derive seconds-per-step from total elapsed time (early steps include one-time warmup
 that can dominate a short run).
 
-> **Even on a genuinely stale heartbeat, do NOT cancel.** Flash **auto-retries a preempted job from its
-> last checkpoint**, and the provider's `job_preempted` notice can lag the freeze by ~10
-> minutes. Cancelling races that retry and kills a run that was recovering on its own.
-> Wait for either the automatic retry or a genuine terminal state. Cancel only when a run
-> is confirmed dead with no retry pending, or is burning money on a definite unrecoverable
-> fault. An idle GPU during model load is also not evidence of a hang — and it is not
-> evidence that your GPU class is unhealthy, so do not "fix" it by pinning another one.
+A stale heartbeat alone does not identify the failure. Check the provider/attempt state and
+wait for a terminal result or a retry decision for a supported failure category. Host preemption
+is terminal: Flash does not automatically restart a preempted job. Preserve any published
+checkpoints for an explicit follow-up run. An idle GPU during model loading can be normal.
 
 ### 6. Deploy & chat
 

freesolo_flash_dev-1.4.7/flash/engine/profiling/sft_tools.py

@@ -0,0 +1,49 @@
+"""tool declarations an SFT row trains against.
+
+A tool-calling agent is served with ``tools=``, and the chat template renders those declarations
+into the prompt (Qwen3.5 prepends a ``# Tools`` block to the system turn). A row rendered without
+them trains on a prompt the deployed adapter never sees. A row may therefore carry ``tools`` -- the
+same per-example field as OpenAI fine-tuning data -- which is validated and normalized exactly as a
+chat request's ``tools`` are (``text_tokenizer`` renders ``tools_wire(normalize_tools(...))``), then
+rendered into every training view of the row. A row without ``tools`` renders as before.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flash.serve.request.tool_calls import normalize_tools, tools_wire
+
+
+def sft_row_tools(example) -> list[dict[str, Any]] | None:
+    """The row's tools in the form serving renders them, or None when it declares none."""
+    if not isinstance(example, dict) or example.get("tools") is None:
+        return None
+    try:
+        return tools_wire(normalize_tools(example["tools"]))
+    except ValueError as exc:
+        raise ValueError(f"sft tools: {exc}") from exc
+
+
+def sft_template_kwargs(thinking: bool, tools: list[dict[str, Any]] | None) -> dict[str, Any]:
+    """Chat-template kwargs for every training view of a row.
+
+    The transcript, the prompt boundary, the reasoning probe and the loss mask's field probe all
+    render with these, so the supervised tokens are located in exactly the text that is trained on.
+    Without tools this is the pre-existing thinking-only render.
+    """
+    kwargs: dict[str, Any] = {"enable_thinking": thinking, "preserve_thinking": False}
+    if tools is not None:
+        kwargs["tools"] = tools
+    return kwargs
+
+
+def reject_image_row_tools(example, *, image_bearing: bool) -> None:
+    """Refuse tools on an image row before any tokenizer or processor is resolved.
+
+    The image pipeline tokenizes through the processor's own render, so accepting them would silently
+    train the row without the declarations it was given. Validating here, while rows are collected,
+    also makes a malformed declaration fail with its own message rather than as a capability error.
+    """
+    if sft_row_tools(example) is not None and image_bearing:
+        raise ValueError("sft tools are not supported on image-bearing rows")

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/engine/profiling/sft_workload.py

@@ -15,6 +15,7 @@ from flash.content.multimodal import NormalizedImages
 from flash.content.thinking import messages_for_chat_template
 from flash.engine.plan.recipe import RECIPE
 from flash.engine.plan.steps import resolve_update_horizon, sft_update_steps
+from flash.engine.profiling import sft_tools
 from flash.engine.profiling.sft_image_rows import (
     estimate_sft_image_row,
     process_sft_image_row,
@@ -417,22 +418,24 @@ def _text_sft_row_spec(
     max_length: int,
     row_index: int,
     multiturn: bool,
+    tools: list[dict[str, Any]] | None = None,
 ) -> tuple[str, _RowReasoning, dict[str, Any]]:
     normalized_prompt = messages_for_chat_template(prompt_messages)
     normalized_completion = messages_for_chat_template(completion_messages)
     normalized_source = [*normalized_prompt, *normalized_completion]
-    text = render_transcript(normalized_source)
+    template_kwargs = sft_tools.sft_template_kwargs(spec.thinking, tools)
+    render = partial(render_transcript, tools=tools)
+    text = render(normalized_source)
     prompt_text = tokenizer.apply_chat_template(
         normalized_prompt,
         tokenize=False,
         add_generation_prompt=True,
-        enable_thinking=spec.thinking,
-        preserve_thinking=False,
+        **template_kwargs,
     )
     reasoning = _row_reasoning(
         prompt_messages,
         completion_messages,
-        render=render_transcript,
+        render=render,
         tokenizer=tokenizer,
         max_length=max_length,
     )
@@ -444,7 +447,7 @@ def _text_sft_row_spec(
             "prompt_text": prompt_text,
             "target_messages": normalized_completion,
             "source_messages": normalized_source,
-            "template_kwargs": {"enable_thinking": spec.thinking, "preserve_thinking": False},
+            "template_kwargs": template_kwargs,
             # measurement carried alongside the row rather than in it: the parquet row is built from
             # an explicit key list below, so neither key reaches verl.
             "row_index": row_index,
@@ -481,13 +484,12 @@ def _tokenize_prompt_rows(
     # denominator. summed after filtering instead.
     reasoning_by_index: dict[int, _RowReasoning] = {}
 
-    def render_transcript(messages: list[dict]) -> str:
+    def render_transcript(messages: list[dict], tools: list[dict[str, Any]] | None = None) -> str:
         return tokenizer.apply_chat_template(
             messages_for_chat_template(messages),
             tokenize=False,
             add_generation_prompt=False,
-            enable_thinking=spec.thinking,
-            preserve_thinking=False,
+            **sft_tools.sft_template_kwargs(spec.thinking, tools),
         )
 
     for row_index, (
@@ -498,6 +500,7 @@ def _tokenize_prompt_rows(
     ) in enumerate(prompt_rows):
         has_images = record_has_images(example, prompt_messages)
         _reject_image_completion(completion_messages, image_bearing=has_images)
+        tools = sft_tools.sft_row_tools(example)
         # read before the image branch rewrites `completion_messages` to its text-only form.
         multiturn = len(completion_messages) > 1
         if multiturn:
@@ -561,6 +564,7 @@ def _tokenize_prompt_rows(
                 max_length=max_length,
                 row_index=row_index,
                 multiturn=multiturn,
+                tools=tools,
             )
             sampled_texts.append(text)
             reasoning_by_index[row_index] = reasoning
@@ -913,6 +917,9 @@ def prepare_sft_workload(
         else:
             prompt_messages = env.prompt_messages(example)
         completion_messages, coerced_scalar_output = _sft_completion_with_provenance(env, example)
+        sft_tools.reject_image_row_tools(
+            example, image_bearing=record_has_images(example, prompt_messages)
+        )
         prompt_rows.append((example, prompt_messages, completion_messages, coerced_scalar_output))
     package_root = getattr(env, "package_root", None)
     multimodal = any(

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/engine/worker/train/entry/backend_common.py

@@ -159,10 +159,7 @@ def _install_flash_attn(py: str) -> None:
             return
         except subprocess.CalledProcessError as e:
             if attempt == FLASH_ATTN_INSTALL_ATTEMPTS - 1:
-                # hand the arm back to the plane as infra-shaped: it retries as job_preempted on a
-                # fresh worker instead of failing the run. bounded by the plane's own retry budget
-                # (INFRA_RETRY_FLOOR), so a genuinely broken wheel spec still terminates rather than
-                # looping -- it just costs a few provisioning attempts to establish that.
+                # preserve the infrastructure failure in the terminal record after local retries.
                 raise RetriableInfraError(
                     f"prebuilt flash-attn wheel install failed {FLASH_ATTN_INSTALL_ATTEMPTS}x "
                     f"(exit {e.returncode}): {FLASH_ATTN_SPEC}"

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/providers/_lifecycle/bootstrapping/bootstrap.py

@@ -56,7 +56,7 @@ _MAX_ATTEMPT_ID = (1 << 63) - 1
 
 
 class RetriableBootstrapError(RuntimeError):
-    """Infra-shaped failure → marker carries retriable=True → poller retries (job_preempted) instead of job_failed."""
+    """Infrastructure failure recorded in the attempt's terminal marker."""
 
 
 class WorkerInterrupted(BaseException):

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/providers/modal/jobs.py

@@ -68,8 +68,8 @@ def poll(handle, spec, *, log=None, deadline_at=None):
 
     def process_failure():
         # the provider exit status survives an artifact-store outage. ordinary process errors
-        # must not become paid preemption retries merely because their failure upload hit 429.
-        # signals/unknown status remain host-loss candidates; explicit retriable markers still win.
+        # must not be mislabeled as host loss merely because their failure upload hit 429.
+        # signals/unknown status remain host-loss candidates; explicit worker markers still win.
         if type(returncode) is int and 0 < returncode < 128:
             return f"Modal worker stopped with exit code {returncode} without a terminal record"
         return None

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/runner/supervise/retry_decision.py

@@ -8,7 +8,7 @@ from dataclasses import dataclass, replace
 from flash.core.spec import JobSpec
 
 INFRA_RETRY_FLOOR = 5
-INFRA_RETRY_FAILURES = frozenset({"stalled", "no_capacity", "poll_error", "job_preempted"})
+INFRA_RETRY_FAILURES = frozenset({"stalled", "no_capacity", "poll_error"})
 CANDIDATE_RETRY_FAILURES = INFRA_RETRY_FAILURES | {"oom"}
 CACHE_FALLBACK_FAILURES = frozenset({"no_capacity", "poll_error"})
 _STATE_KEYS = frozenset(
@@ -185,7 +185,11 @@ def _restore_last_decision(raw: object, counters: dict) -> PersistedRetryDecisio
         or not failure
     ):
         raise ValueError("persisted retry state has an invalid last decision")
-    return PersistedRetryDecision(attempt, failure, _restore_plan(raw["plan"], counters))
+    plan = _restore_plan(raw["plan"], counters)
+    # an older saved decision cannot authorize a failure the current policy no longer retries.
+    if plan.retry and failure not in CANDIDATE_RETRY_FAILURES:
+        plan = RetryPlan(False, "not retrying")
+    return PersistedRetryDecision(attempt, failure, plan)
 
 
 def _decision_snapshot(decision: PersistedRetryDecision | None) -> dict | None:
@@ -325,20 +329,6 @@ def transition_failure(
             observation,
             attempt,
         )
-    if failure == "job_preempted":
-        # Preemption says nothing about memory demand. Advancing the OOM floor here
-        # made a healthy worker at the largest allowed shape impossible to resume.
-        if state.infra_used >= state.infra_retries:
-            plan = RetryPlan(False, "not retrying: infrastructure retry budget exhausted")
-        else:
-            ordinal = state.infra_used + 1
-            state = replace(state, infra_used=ordinal, cache_retry_shape=None)
-            plan = RetryPlan(
-                True,
-                "retrying after preemption (resume from last checkpoint)",
-                ordinal,
-            )
-        return _finish(state, plan, observation, attempt)
     floor = max(state.usable_vram_floor, _candidate_usable_vram_gb(chosen))
     state = replace(state, usable_vram_floor=floor, cache_retry_shape=None)
     if candidates is not None and not any(

freesolo_flash_dev-1.4.7/flash/serve/request/schema_types.py

@@ -0,0 +1,36 @@
+"""Exact JSON Schema type matching for tool declarations and arguments."""
+
+import math
+from decimal import Decimal
+from typing import Any
+
+
+def _matches_type(value: Any, schema_type: str | list[str]) -> bool:
+    if type(schema_type) is list:
+        return any(_matches_type(value, member) for member in schema_type)
+    if schema_type == "null":
+        return value is None
+    if schema_type == "boolean":
+        return type(value) is bool
+    if schema_type == "integer":
+        return (
+            type(value) is int
+            or (type(value) is Decimal and _decimal_is_integral(value))
+            or (type(value) is float and math.isfinite(value) and value.is_integer())
+        )
+    if schema_type == "number":
+        if type(value) is Decimal:
+            return value.is_finite()
+        return type(value) is int or (type(value) is float and math.isfinite(value))
+    if schema_type == "string":
+        return type(value) is str
+    if schema_type == "array":
+        return type(value) is list
+    return type(value) is dict
+
+
+def _decimal_is_integral(value: Decimal) -> bool:
+    if not value.is_finite():
+        return False
+    digits, exponent = value.as_tuple().digits, value.as_tuple().exponent
+    return exponent >= 0 or all(digit == 0 for digit in digits[exponent:])

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/serve/request/tool_calls.py

@@ -12,6 +12,7 @@ from typing import Any
 
 from flash.serve.contract.protocol import MAX_CHAT_REQUEST_BYTES, TEXT_TYPES
 from flash.serve.request import text_scan
+from flash.serve.request.schema_types import _decimal_is_integral, _matches_type
 from flash.serve.request.text_scan import TOOL_CALL_END, TOOL_CALL_START
 from flash.serve.request.text_scan import skip_whitespace as _skip_whitespace
 from flash.serve.request.tool_template import last_query_index, rendered_turn_prefix
@@ -37,6 +38,7 @@ _MAX_FIXED_DECIMAL_DIGITS = _MAX_NUMERIC_LITERAL_DIGITS = 1024
 _MAX_NUMERIC_LITERAL_EXPONENT = 1_000_000
 _MAX_ENUM_INTEGER_MAGNITUDE = 10**_MAX_NUMERIC_LITERAL_DIGITS
 _SCHEMA_TYPES = frozenset(["array", "boolean", "integer", "null", "number", "object", "string"])  # fmt: skip
+_SCALAR_SCHEMA_TYPES = _SCHEMA_TYPES - {"array", "object"}
 _SCHEMA_KEYS = frozenset(["additionalProperties", "description", "enum", "items", "properties", "required", "type"])  # fmt: skip
 
 
@@ -317,7 +319,22 @@ def _normalize_schema(
     if unknown:
         raise error_type(f"{path} uses unsupported schema keyword(s): {', '.join(sorted(unknown))}")
     schema_type = raw.get("type")
-    if type(schema_type) is not str or schema_type not in _SCHEMA_TYPES:
+    if type(schema_type) is list:
+        if (
+            not 1 <= len(schema_type) <= len(_SCALAR_SCHEMA_TYPES)
+            or any(
+                type(member) is not str or member not in _SCALAR_SCHEMA_TYPES
+                for member in schema_type
+            )
+            or len(set(schema_type)) != len(schema_type)
+        ):
+            raise error_type(
+                f"{path}.type must be a nonempty unique list of supported scalar types"
+            )
+        if root or direct:
+            raise error_type(f"{path}.type unions are supported only inside JSON arrays or objects")
+        schema_type = list(schema_type)
+    elif type(schema_type) is not str or schema_type not in _SCHEMA_TYPES:
         raise error_type(f"{path}.type must be one supported JSON Schema type")
     if root and schema_type != "object":
         raise error_type(f"{path} must be a root object schema")
@@ -498,7 +515,8 @@ def _validate_tool_result(
 
 def _canonicalize_integer_values(value: Any, schema: Mapping[str, Any]) -> Any:
     schema_type = schema["type"]
-    if schema_type == "integer" and type(value) is Decimal:
+    types = schema_type if type(schema_type) is list else [schema_type]
+    if "integer" in types and "number" not in types and type(value) is Decimal:
         return value.to_integral_value()
     if schema_type == "object":
         properties = schema["properties"]
@@ -511,35 +529,6 @@ def _canonicalize_integer_values(value: Any, schema: Mapping[str, Any]) -> Any:
     return value
 
 
-def _matches_type(value: Any, schema_type: str) -> bool:
-    if schema_type == "null":
-        return value is None
-    if schema_type == "boolean":
-        return type(value) is bool
-    if schema_type == "integer":
-        return (
-            type(value) is int
-            or (type(value) is Decimal and _decimal_is_integral(value))
-            or (type(value) is float and math.isfinite(value) and value.is_integer())
-        )
-    if schema_type == "number":
-        if type(value) is Decimal:
-            return value.is_finite()
-        return type(value) is int or (type(value) is float and math.isfinite(value))
-    if schema_type == "string":
-        return type(value) is str
-    if schema_type == "array":
-        return type(value) is list
-    return type(value) is dict
-
-
-def _decimal_is_integral(value: Decimal) -> bool:
-    if not value.is_finite():
-        return False
-    digits, exponent = value.as_tuple().digits, value.as_tuple().exponent
-    return exponent >= 0 or all(digit == 0 for digit in digits[exponent:])
-
-
 def _load_exact_json(value: str) -> Any:
     try:
         decoded = json.loads(

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/serve/runtime/tool_calls.py

@@ -660,7 +660,7 @@ def _validate_value(value: Any, schema: Mapping[str, Any]) -> bool:
         return type(value) in {dict, list}
     if not _matches_type(value, schema_type):
         return False
-    if schema_type == "string" and _contains_unpaired_surrogate(value):
+    if type(value) is str and _contains_unpaired_surrogate(value):
         return False
     if "enum" in schema and not any(_json_values_equal(value, item) for item in schema["enum"]):
         return False

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/server/asgi/app.py

@@ -198,16 +198,18 @@ async def _lead_when_free(work: _LeaderWork) -> None:
     work.recovery = asyncio.ensure_future(asyncio.to_thread(work.recover))
     try:
         await asyncio.shield(work.recovery)
+        # Starting the loops can fail too, e.g. the deployment-job pool is full of jobs the plane
+        # accepted in standby. Leave standby only once the holder's work is actually running.
+        work.start()
     except asyncio.CancelledError:
         raise
     except Exception:
-        # A lone plane fails startup when recovery fails; a takeover must not keep serving as if
-        # it led. Stop, and let the restart recover from a clean start.
-        _log.critical("control-plane recovery failed after lease takeover; stopping", exc_info=True)
+        # A lone plane fails startup when recovery or its loops fail; a takeover must not keep
+        # serving as if it led. Stop, and let the restart recover from a clean start.
+        _log.critical("control-plane takeover failed; stopping", exc_info=True)
         _stop_plane()
         return
     work.app.state.plane_standby = False
-    work.start()
 
 
 def create_app():

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/server/platform/auth.py

@@ -31,6 +31,12 @@ _verify_cache: dict[str, float] = {}
 _verify_cache_lock = threading.Lock()
 _verify_inflight: dict[str, Future[dict[str, Any] | None]] = {}
 _VERIFY_CACHE_MAX = 1024
+# token digest -> (monotonic verified-at, identity) of the last successful verification. it is never
+# served while the identity service answers, so revocation stays immediate; it only bridges brief
+# identity-service outages for keys verified moments ago, which otherwise fail every request with a
+# 503 (e.g. thousands of rollout chat calls). a definitive rejection forgets it.
+_VERIFY_STALE_GRACE_S = 300.0
+_verify_last_good: dict[str, tuple[float, dict[str, Any]]] = {}
 
 
 class AuthVerificationUnavailable(RuntimeError):
@@ -53,6 +59,27 @@ def _prune_verify_cache_locked(now: float) -> None:
             del _verify_cache[digest]
 
 
+def _remember_good_locked(digest: str, identity: dict[str, Any], now: float) -> None:
+    """Record a successful verification. Caller must hold ``_verify_cache_lock``."""
+    for stale in [
+        d for d, (at, _) in _verify_last_good.items() if now - at > _VERIFY_STALE_GRACE_S
+    ]:
+        del _verify_last_good[stale]
+    if digest not in _verify_last_good and len(_verify_last_good) >= _VERIFY_CACHE_MAX:
+        oldest = min(_verify_last_good, key=lambda d: _verify_last_good[d][0])
+        del _verify_last_good[oldest]
+    _verify_last_good[digest] = (now, dict(identity))
+
+
+def _recent_good_identity(digest: str) -> dict[str, Any] | None:
+    """Identity verified within the grace window, for use only when verification is unavailable."""
+    with _verify_cache_lock:
+        entry = _verify_last_good.get(digest)
+        if entry is None or time.monotonic() - entry[0] > _VERIFY_STALE_GRACE_S:
+            return None
+        return dict(entry[1])
+
+
 def _freesolo_key_prefix(token: str) -> str:
     """Non-secret preview of a Freesolo API key, matching the dashboard-style public prefix."""
     parts = token.split("_", 2)
@@ -232,6 +259,22 @@ def _freesolo_verify(token: str) -> dict[str, Any] | None:
                 now = time.monotonic()
                 _prune_verify_cache_locked(now)
                 _verify_cache[digest] = now + _VERIFY_CACHE_NEG_TTL_S
+                _verify_last_good.pop(digest, None)
+        elif identity is not None:
+            with _verify_cache_lock:
+                _remember_good_locked(digest, identity, time.monotonic())
+    except AuthVerificationUnavailable as exc:
+        fallback = _recent_good_identity(digest)
+        with _verify_cache_lock:
+            if _verify_inflight.get(digest) is pending:
+                del _verify_inflight[digest]
+            if fallback is None:
+                pending.set_exception(exc)
+            else:
+                pending.set_result(fallback)
+        if fallback is None:
+            raise
+        return fallback
     except BaseException as exc:
         with _verify_cache_lock:
             if _verify_inflight.get(digest) is pending:

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/flash/serving/deepinfer/app/modal_app.py

@@ -68,6 +68,7 @@ if main_environment:
         schedule=modal.Period(seconds=POOL_SAMPLE_SECONDS),
         cpu=0.25,
         memory=256,
+        nonpreemptible=True,
         min_containers=1,
         max_containers=1,
         timeout=15,

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "freesolo-flash-dev"
-version = "1.4.5"
+version = "1.4.7"
 description ="Flash — managed LoRA post-training (SFT/GRPO) for Freesolo environments, driven by the `flash` CLI"
 readme = "README.md"
 # The training and serving stack is validated on Python 3.11 and 3.12.
@@ -209,7 +209,7 @@ artifacts = ["flash/py.typed"]
 # auto-published by .github/workflows/publish-dev.yml: a push to `dev` whose version isn't yet on
 # PyPI publishes it; pushes that don't change it find it already published and no-op. So cutting a
 # dev-channel release = bump this (and [project].version) and merge to `dev`.
-version = "1.4.5"
+version = "1.4.7"
 
 [tool.pytest.ini_options]
 pythonpath = ["."]

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/tests/serving/test_deepinfer_autoscaling.py

@@ -1,6 +1,8 @@
 """Pool thresholds, telemetry failures, and complete streaming request lifetimes."""
 
 import asyncio
+import runpy
+from pathlib import Path
 from types import SimpleNamespace
 
 import pytest
@@ -235,3 +237,14 @@ def test_counter_covers_preparation_and_stream_lifetime(path, finish):
         assert counts["active"] == 0
 
     asyncio.run(run())
+
+
+def test_deployed_pool_controller_uses_nonpreemptible_cpu_capacity(monkeypatch):
+    monkeypatch.setenv("MODAL_ENVIRONMENT", "main")
+    module = runpy.run_path(
+        str(Path(__file__).parents[2] / "flash/serving/deepinfer/app/modal_app.py")
+    )
+    info = module["scale_pool"].info()
+    assert info.nonpreemptible
+    assert not info.gpus
+    assert info.schedule is not None

{freesolo_flash_dev-1.4.5 → freesolo_flash_dev-1.4.7}/tests/test_plane_lease.py

@@ -192,6 +192,28 @@ def test_takeover_recovery_failure_stops_the_plane_instead_of_serving_as_leader(
         assert loops == []
 
 
+def test_takeover_that_cannot_start_its_loops_stops_the_plane(monkeypatch):
+    """Recovery succeeding is not enough: e.g. a full deployment-job pool fails the loops' start."""
+    import flash.server.asgi.app as app_mod
+
+    recovered, stopped = [], threading.Event()
+    _stub_lifespan(monkeypatch, recovered)
+
+    def pool_full(self):
+        raise RuntimeError("deployment job capacity reached")
+
+    monkeypatch.setattr(app_mod, "_stop_plane", stopped.set)
+    monkeypatch.setattr(app_mod._LeaderWork, "start", pool_full)
+    old_plane = _hold_elsewhere()
+
+    with TestClient(app_mod.create_app()) as client:
+        _release_elsewhere(old_plane)
+        assert stopped.wait(10)
+        assert recovered == ["runs", "deployments"]
+        # never leaves standby, so submissions are not launched by a plane without its loops.
+        assert client.app.state.plane_standby is True
+
+
 def test_shutdown_waits_for_takeover_recovery_before_releasing_the_lease(monkeypatch):
     import flash.server.asgi.app as app_mod
     import flash.server.platform.runtime as server_runtime