freesolo-flash-dev 1.4.6__tar.gz → 1.4.7__tar.gz

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/CHANGELOG.md

@@ -162,6 +162,12 @@ This file starts at 1.1.35. Earlier releases are not reconstructed here; use
 
 ### Fixed
 
+- API-key verification bridges brief Freesolo identity-service outages. A key that verified
+  successfully within the last 5 minutes is accepted when verification is unavailable (transport
+  errors or 5xx), instead of failing the request with `503 Authentication service unavailable`.
+  Live verification still runs on every request while the service answers, and a definitive
+  rejection forgets the key, so revocation stays immediate. Long multi-turn rollouts previously
+  lost about 13% of episodes to these 503s.
 - Image SFT no longer stores processed pixel tensors in the training parquet. The worker keeps
   each row's small processor fields (image grid, token types) and its saved image files, and the
   training dataset rebuilds `pixel_values` per item, checking the stored grid. Before this,

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: freesolo-flash-dev
-Version: 1.4.6
+Version: 1.4.7
 Summary: Flash — managed LoRA post-training (SFT/GRPO) for Freesolo environments, driven by the `flash` CLI
 Project-URL: Homepage, https://github.com/freesolo-co/flash
 Project-URL: Repository, https://github.com/freesolo-co/flash
@@ -341,6 +341,13 @@ endpoint returns `409` with a hint to deploy first. Prefer this control-plane en
 directly: it enforces run ownership and forwards per-run serving options such as
 thinking-mode parity.
 
+Tool parameters use a restricted JSON Schema profile. Inside JSON arrays and objects,
+scalar types can be combined explicitly, for example `"type": ["string", "number",
+"boolean", "null"]` for spreadsheet cells. Empty schemas (`{}`), container unions,
+and unions on top-level function arguments are unsupported. Top-level arguments use
+native parameter text, where a string such as `"12"` and a number would be ambiguous;
+nested JSON preserves those types during generation, streaming, and history replay.
+
 ### Workload estimates (SFT)
 
 `flash train`, `--dry-run`, and `--cost` read the pinned environment's packaged dataset

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/README.md

@@ -241,6 +241,13 @@ endpoint returns `409` with a hint to deploy first. Prefer this control-plane en
 directly: it enforces run ownership and forwards per-run serving options such as
 thinking-mode parity.
 
+Tool parameters use a restricted JSON Schema profile. Inside JSON arrays and objects,
+scalar types can be combined explicitly, for example `"type": ["string", "number",
+"boolean", "null"]` for spreadsheet cells. Empty schemas (`{}`), container unions,
+and unions on top-level function arguments are unsupported. Top-level arguments use
+native parameter text, where a string such as `"12"` and a number would be ambiguous;
+nested JSON preserves those types during generation, streaming, and history replay.
+
 ### Workload estimates (SFT)
 
 `flash train`, `--dry-run`, and `--cost` read the pinned environment's packaged dataset

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/cli/scaffold/TRAINING.md

@@ -461,13 +461,10 @@ described above.
 Never derive seconds-per-step from total elapsed time (early steps include one-time warmup
 that can dominate a short run).
 
-> **Even on a genuinely stale heartbeat, do NOT cancel.** Flash **auto-retries a preempted job from its
-> last checkpoint**, and the provider's `job_preempted` notice can lag the freeze by ~10
-> minutes. Cancelling races that retry and kills a run that was recovering on its own.
-> Wait for either the automatic retry or a genuine terminal state. Cancel only when a run
-> is confirmed dead with no retry pending, or is burning money on a definite unrecoverable
-> fault. An idle GPU during model load is also not evidence of a hang — and it is not
-> evidence that your GPU class is unhealthy, so do not "fix" it by pinning another one.
+A stale heartbeat alone does not identify the failure. Check the provider/attempt state and
+wait for a terminal result or a retry decision for a supported failure category. Host preemption
+is terminal: Flash does not automatically restart a preempted job. Preserve any published
+checkpoints for an explicit follow-up run. An idle GPU during model loading can be normal.
 
 ### 6. Deploy & chat
 

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/engine/worker/train/entry/backend_common.py

@@ -159,10 +159,7 @@ def _install_flash_attn(py: str) -> None:
             return
         except subprocess.CalledProcessError as e:
             if attempt == FLASH_ATTN_INSTALL_ATTEMPTS - 1:
-                # hand the arm back to the plane as infra-shaped: it retries as job_preempted on a
-                # fresh worker instead of failing the run. bounded by the plane's own retry budget
-                # (INFRA_RETRY_FLOOR), so a genuinely broken wheel spec still terminates rather than
-                # looping -- it just costs a few provisioning attempts to establish that.
+                # preserve the infrastructure failure in the terminal record after local retries.
                 raise RetriableInfraError(
                     f"prebuilt flash-attn wheel install failed {FLASH_ATTN_INSTALL_ATTEMPTS}x "
                     f"(exit {e.returncode}): {FLASH_ATTN_SPEC}"

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/providers/_lifecycle/bootstrapping/bootstrap.py

@@ -56,7 +56,7 @@ _MAX_ATTEMPT_ID = (1 << 63) - 1
 
 
 class RetriableBootstrapError(RuntimeError):
-    """Infra-shaped failure → marker carries retriable=True → poller retries (job_preempted) instead of job_failed."""
+    """Infrastructure failure recorded in the attempt's terminal marker."""
 
 
 class WorkerInterrupted(BaseException):

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/providers/modal/jobs.py

@@ -68,8 +68,8 @@ def poll(handle, spec, *, log=None, deadline_at=None):
 
     def process_failure():
         # the provider exit status survives an artifact-store outage. ordinary process errors
-        # must not become paid preemption retries merely because their failure upload hit 429.
-        # signals/unknown status remain host-loss candidates; explicit retriable markers still win.
+        # must not be mislabeled as host loss merely because their failure upload hit 429.
+        # signals/unknown status remain host-loss candidates; explicit worker markers still win.
         if type(returncode) is int and 0 < returncode < 128:
             return f"Modal worker stopped with exit code {returncode} without a terminal record"
         return None

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/runner/supervise/retry_decision.py

@@ -8,7 +8,7 @@ from dataclasses import dataclass, replace
 from flash.core.spec import JobSpec
 
 INFRA_RETRY_FLOOR = 5
-INFRA_RETRY_FAILURES = frozenset({"stalled", "no_capacity", "poll_error", "job_preempted"})
+INFRA_RETRY_FAILURES = frozenset({"stalled", "no_capacity", "poll_error"})
 CANDIDATE_RETRY_FAILURES = INFRA_RETRY_FAILURES | {"oom"}
 CACHE_FALLBACK_FAILURES = frozenset({"no_capacity", "poll_error"})
 _STATE_KEYS = frozenset(
@@ -185,7 +185,11 @@ def _restore_last_decision(raw: object, counters: dict) -> PersistedRetryDecisio
         or not failure
     ):
         raise ValueError("persisted retry state has an invalid last decision")
-    return PersistedRetryDecision(attempt, failure, _restore_plan(raw["plan"], counters))
+    plan = _restore_plan(raw["plan"], counters)
+    # an older saved decision cannot authorize a failure the current policy no longer retries.
+    if plan.retry and failure not in CANDIDATE_RETRY_FAILURES:
+        plan = RetryPlan(False, "not retrying")
+    return PersistedRetryDecision(attempt, failure, plan)
 
 
 def _decision_snapshot(decision: PersistedRetryDecision | None) -> dict | None:
@@ -325,20 +329,6 @@ def transition_failure(
             observation,
             attempt,
         )
-    if failure == "job_preempted":
-        # Preemption says nothing about memory demand. Advancing the OOM floor here
-        # made a healthy worker at the largest allowed shape impossible to resume.
-        if state.infra_used >= state.infra_retries:
-            plan = RetryPlan(False, "not retrying: infrastructure retry budget exhausted")
-        else:
-            ordinal = state.infra_used + 1
-            state = replace(state, infra_used=ordinal, cache_retry_shape=None)
-            plan = RetryPlan(
-                True,
-                "retrying after preemption (resume from last checkpoint)",
-                ordinal,
-            )
-        return _finish(state, plan, observation, attempt)
     floor = max(state.usable_vram_floor, _candidate_usable_vram_gb(chosen))
     state = replace(state, usable_vram_floor=floor, cache_retry_shape=None)
     if candidates is not None and not any(

freesolo_flash_dev-1.4.7/flash/serve/request/schema_types.py

@@ -0,0 +1,36 @@
+"""Exact JSON Schema type matching for tool declarations and arguments."""
+
+import math
+from decimal import Decimal
+from typing import Any
+
+
+def _matches_type(value: Any, schema_type: str | list[str]) -> bool:
+    if type(schema_type) is list:
+        return any(_matches_type(value, member) for member in schema_type)
+    if schema_type == "null":
+        return value is None
+    if schema_type == "boolean":
+        return type(value) is bool
+    if schema_type == "integer":
+        return (
+            type(value) is int
+            or (type(value) is Decimal and _decimal_is_integral(value))
+            or (type(value) is float and math.isfinite(value) and value.is_integer())
+        )
+    if schema_type == "number":
+        if type(value) is Decimal:
+            return value.is_finite()
+        return type(value) is int or (type(value) is float and math.isfinite(value))
+    if schema_type == "string":
+        return type(value) is str
+    if schema_type == "array":
+        return type(value) is list
+    return type(value) is dict
+
+
+def _decimal_is_integral(value: Decimal) -> bool:
+    if not value.is_finite():
+        return False
+    digits, exponent = value.as_tuple().digits, value.as_tuple().exponent
+    return exponent >= 0 or all(digit == 0 for digit in digits[exponent:])

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/serve/request/tool_calls.py

@@ -12,6 +12,7 @@ from typing import Any
 
 from flash.serve.contract.protocol import MAX_CHAT_REQUEST_BYTES, TEXT_TYPES
 from flash.serve.request import text_scan
+from flash.serve.request.schema_types import _decimal_is_integral, _matches_type
 from flash.serve.request.text_scan import TOOL_CALL_END, TOOL_CALL_START
 from flash.serve.request.text_scan import skip_whitespace as _skip_whitespace
 from flash.serve.request.tool_template import last_query_index, rendered_turn_prefix
@@ -37,6 +38,7 @@ _MAX_FIXED_DECIMAL_DIGITS = _MAX_NUMERIC_LITERAL_DIGITS = 1024
 _MAX_NUMERIC_LITERAL_EXPONENT = 1_000_000
 _MAX_ENUM_INTEGER_MAGNITUDE = 10**_MAX_NUMERIC_LITERAL_DIGITS
 _SCHEMA_TYPES = frozenset(["array", "boolean", "integer", "null", "number", "object", "string"])  # fmt: skip
+_SCALAR_SCHEMA_TYPES = _SCHEMA_TYPES - {"array", "object"}
 _SCHEMA_KEYS = frozenset(["additionalProperties", "description", "enum", "items", "properties", "required", "type"])  # fmt: skip
 
 
@@ -317,7 +319,22 @@ def _normalize_schema(
     if unknown:
         raise error_type(f"{path} uses unsupported schema keyword(s): {', '.join(sorted(unknown))}")
     schema_type = raw.get("type")
-    if type(schema_type) is not str or schema_type not in _SCHEMA_TYPES:
+    if type(schema_type) is list:
+        if (
+            not 1 <= len(schema_type) <= len(_SCALAR_SCHEMA_TYPES)
+            or any(
+                type(member) is not str or member not in _SCALAR_SCHEMA_TYPES
+                for member in schema_type
+            )
+            or len(set(schema_type)) != len(schema_type)
+        ):
+            raise error_type(
+                f"{path}.type must be a nonempty unique list of supported scalar types"
+            )
+        if root or direct:
+            raise error_type(f"{path}.type unions are supported only inside JSON arrays or objects")
+        schema_type = list(schema_type)
+    elif type(schema_type) is not str or schema_type not in _SCHEMA_TYPES:
         raise error_type(f"{path}.type must be one supported JSON Schema type")
     if root and schema_type != "object":
         raise error_type(f"{path} must be a root object schema")
@@ -498,7 +515,8 @@ def _validate_tool_result(
 
 def _canonicalize_integer_values(value: Any, schema: Mapping[str, Any]) -> Any:
     schema_type = schema["type"]
-    if schema_type == "integer" and type(value) is Decimal:
+    types = schema_type if type(schema_type) is list else [schema_type]
+    if "integer" in types and "number" not in types and type(value) is Decimal:
         return value.to_integral_value()
     if schema_type == "object":
         properties = schema["properties"]
@@ -511,35 +529,6 @@ def _canonicalize_integer_values(value: Any, schema: Mapping[str, Any]) -> Any:
     return value
 
 
-def _matches_type(value: Any, schema_type: str) -> bool:
-    if schema_type == "null":
-        return value is None
-    if schema_type == "boolean":
-        return type(value) is bool
-    if schema_type == "integer":
-        return (
-            type(value) is int
-            or (type(value) is Decimal and _decimal_is_integral(value))
-            or (type(value) is float and math.isfinite(value) and value.is_integer())
-        )
-    if schema_type == "number":
-        if type(value) is Decimal:
-            return value.is_finite()
-        return type(value) is int or (type(value) is float and math.isfinite(value))
-    if schema_type == "string":
-        return type(value) is str
-    if schema_type == "array":
-        return type(value) is list
-    return type(value) is dict
-
-
-def _decimal_is_integral(value: Decimal) -> bool:
-    if not value.is_finite():
-        return False
-    digits, exponent = value.as_tuple().digits, value.as_tuple().exponent
-    return exponent >= 0 or all(digit == 0 for digit in digits[exponent:])
-
-
 def _load_exact_json(value: str) -> Any:
     try:
         decoded = json.loads(

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/serve/runtime/tool_calls.py

@@ -660,7 +660,7 @@ def _validate_value(value: Any, schema: Mapping[str, Any]) -> bool:
         return type(value) in {dict, list}
     if not _matches_type(value, schema_type):
         return False
-    if schema_type == "string" and _contains_unpaired_surrogate(value):
+    if type(value) is str and _contains_unpaired_surrogate(value):
         return False
     if "enum" in schema and not any(_json_values_equal(value, item) for item in schema["enum"]):
         return False

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/server/platform/auth.py

@@ -31,6 +31,12 @@ _verify_cache: dict[str, float] = {}
 _verify_cache_lock = threading.Lock()
 _verify_inflight: dict[str, Future[dict[str, Any] | None]] = {}
 _VERIFY_CACHE_MAX = 1024
+# token digest -> (monotonic verified-at, identity) of the last successful verification. it is never
+# served while the identity service answers, so revocation stays immediate; it only bridges brief
+# identity-service outages for keys verified moments ago, which otherwise fail every request with a
+# 503 (e.g. thousands of rollout chat calls). a definitive rejection forgets it.
+_VERIFY_STALE_GRACE_S = 300.0
+_verify_last_good: dict[str, tuple[float, dict[str, Any]]] = {}
 
 
 class AuthVerificationUnavailable(RuntimeError):
@@ -53,6 +59,27 @@ def _prune_verify_cache_locked(now: float) -> None:
             del _verify_cache[digest]
 
 
+def _remember_good_locked(digest: str, identity: dict[str, Any], now: float) -> None:
+    """Record a successful verification. Caller must hold ``_verify_cache_lock``."""
+    for stale in [
+        d for d, (at, _) in _verify_last_good.items() if now - at > _VERIFY_STALE_GRACE_S
+    ]:
+        del _verify_last_good[stale]
+    if digest not in _verify_last_good and len(_verify_last_good) >= _VERIFY_CACHE_MAX:
+        oldest = min(_verify_last_good, key=lambda d: _verify_last_good[d][0])
+        del _verify_last_good[oldest]
+    _verify_last_good[digest] = (now, dict(identity))
+
+
+def _recent_good_identity(digest: str) -> dict[str, Any] | None:
+    """Identity verified within the grace window, for use only when verification is unavailable."""
+    with _verify_cache_lock:
+        entry = _verify_last_good.get(digest)
+        if entry is None or time.monotonic() - entry[0] > _VERIFY_STALE_GRACE_S:
+            return None
+        return dict(entry[1])
+
+
 def _freesolo_key_prefix(token: str) -> str:
     """Non-secret preview of a Freesolo API key, matching the dashboard-style public prefix."""
     parts = token.split("_", 2)
@@ -232,6 +259,22 @@ def _freesolo_verify(token: str) -> dict[str, Any] | None:
                 now = time.monotonic()
                 _prune_verify_cache_locked(now)
                 _verify_cache[digest] = now + _VERIFY_CACHE_NEG_TTL_S
+                _verify_last_good.pop(digest, None)
+        elif identity is not None:
+            with _verify_cache_lock:
+                _remember_good_locked(digest, identity, time.monotonic())
+    except AuthVerificationUnavailable as exc:
+        fallback = _recent_good_identity(digest)
+        with _verify_cache_lock:
+            if _verify_inflight.get(digest) is pending:
+                del _verify_inflight[digest]
+            if fallback is None:
+                pending.set_exception(exc)
+            else:
+                pending.set_result(fallback)
+        if fallback is None:
+            raise
+        return fallback
     except BaseException as exc:
         with _verify_cache_lock:
             if _verify_inflight.get(digest) is pending:

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/flash/serving/deepinfer/app/modal_app.py

@@ -68,6 +68,7 @@ if main_environment:
         schedule=modal.Period(seconds=POOL_SAMPLE_SECONDS),
         cpu=0.25,
         memory=256,
+        nonpreemptible=True,
         min_containers=1,
         max_containers=1,
         timeout=15,

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "freesolo-flash-dev"
-version = "1.4.6"
+version = "1.4.7"
 description ="Flash — managed LoRA post-training (SFT/GRPO) for Freesolo environments, driven by the `flash` CLI"
 readme = "README.md"
 # The training and serving stack is validated on Python 3.11 and 3.12.
@@ -209,7 +209,7 @@ artifacts = ["flash/py.typed"]
 # auto-published by .github/workflows/publish-dev.yml: a push to `dev` whose version isn't yet on
 # PyPI publishes it; pushes that don't change it find it already published and no-op. So cutting a
 # dev-channel release = bump this (and [project].version) and merge to `dev`.
-version = "1.4.6"
+version = "1.4.7"
 
 [tool.pytest.ini_options]
 pythonpath = ["."]

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/tests/serving/test_deepinfer_autoscaling.py

@@ -1,6 +1,8 @@
 """Pool thresholds, telemetry failures, and complete streaming request lifetimes."""
 
 import asyncio
+import runpy
+from pathlib import Path
 from types import SimpleNamespace
 
 import pytest
@@ -235,3 +237,14 @@ def test_counter_covers_preparation_and_stream_lifetime(path, finish):
         assert counts["active"] == 0
 
     asyncio.run(run())
+
+
+def test_deployed_pool_controller_uses_nonpreemptible_cpu_capacity(monkeypatch):
+    monkeypatch.setenv("MODAL_ENVIRONMENT", "main")
+    module = runpy.run_path(
+        str(Path(__file__).parents[2] / "flash/serving/deepinfer/app/modal_app.py")
+    )
+    info = module["scale_pool"].info()
+    assert info.nonpreemptible
+    assert not info.gpus
+    assert info.schedule is not None

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/tests/test_resume_on_retry.py

@@ -1,15 +1,7 @@
-"""Resume-from-latest-checkpoint on crash/preemption retry (CPU-only).
+"""Checkpoint resume for supported infrastructure retries; all network/GPU boundaries are stubbed.
 
-A crash- or preemption-killed run is relaunched on a fresh worker by the control-plane
-retry loop, and that fresh worker continues training from the latest streamed checkpoint
-rather than restarting from scratch. This file locks in the two live load-bearing pieces:
-
-  1. worker resume: ``hf_resume_checkpoint`` pulls the stream and returns the highest-step
-     checkpoint directory used by the training backend.
-  2. control plane: ``_run_attempts_supervised`` relaunches on the same run id and seed for every
-     infra-shaped failure; a genuine worker error fails fast instead.
-
-All HF/provider/network boundaries are stubbed; nothing here touches a GPU or the network.
+Host preemption is terminal. Checkpoint loading remains necessary for the supervisor's other
+retry categories and explicit checkpoint-based training.
 """
 
 from __future__ import annotations
@@ -30,9 +22,6 @@ from flash.core.spec import JobSpec
 from tests._helpers.profile import attach_sft_profile, stub_revision_geometry
 from tests._helpers.source_snapshot import valid_source_snapshot
 
-# Infra-shaped failure categories the retry loop resumes on (see lifecycle._run_attempts_supervised).
-# Mirrors the literal tuple in the source; this test is the guard that the set doesn't silently drift.
-INFRA_SHAPED = ("stalled", "no_capacity", "poll_error", "job_preempted")
 _SOURCE_SNAPSHOT = valid_source_snapshot()
 
 

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/tests/test_retry_decision.py

@@ -221,28 +221,34 @@ def test_atomic_retry_cas_has_one_owner_and_reuses_persisted_plan(monkeypatch, t
     assert persisted.last_decision.failure == "stalled"
 
 
-def test_preemption_resumes_at_the_largest_shape_without_raising_the_oom_floor():
-    chosen = _candidate("modal", "H100", 80, 8)
-    original = replace(RetryState.initial_for_spec(_spec()), usable_vram_floor=40)
+@pytest.mark.parametrize("budget", [0, 2, 5])
+def test_preemption_is_terminal_without_spending_retry_budget(budget):
+    chosen = _candidate("modal", "L4", 24)
+    larger = _candidate("modal", "H100", 80)
+    original = replace(RetryState(budget, 2, 1), usable_vram_floor=12)
     updated, plan = transition_failure(
-        original, _observe("job_preempted", chosen, (chosen,)), attempt=0
+        original, _observe("job_preempted", chosen, (chosen, larger)), attempt=0
     )
-    restored = RetryState.from_snapshot(_spec(), updated.to_snapshot())
-    assert plan.retry
-    assert plan.infra_retry_ordinal == 1
-    assert restored.usable_vram_floor == 40
-    assert restored.oom_used == 0
-    assert restored.select_candidate((chosen,)) == ((chosen,), chosen)
-
-
-@pytest.mark.parametrize("budget", [0, 2])
-def test_preemption_retries_are_bounded_even_when_the_same_shape_is_available(budget):
-    chosen = _candidate("modal", "H100", 80, 8)
-    retry_state = RetryState(budget, 2, 1)
-    for attempt in range(budget + 1):
-        retry_state, plan = transition_failure(
-            retry_state, _observe("job_preempted", chosen, (chosen,)), attempt=attempt
-        )
-        assert plan.retry is (attempt < budget)
-    assert retry_state.infra_used == budget
-    assert "budget exhausted" in plan.action
+    assert not plan.retry
+    assert updated.infra_used == original.infra_used
+    assert updated.oom_used == original.oom_used
+    assert updated.usable_vram_floor == original.usable_vram_floor
+    assert updated.last_decision.failure == "job_preempted"
+
+
+def test_saved_preemption_retry_cannot_authorize_a_replacement_worker():
+    from flash.runner.supervise.retry_decision import require_retry_authorization_from_raw
+
+    spec = _spec()
+    old_state = replace(
+        RetryState.initial_for_spec(spec),
+        infra_used=1,
+        last_decision=PersistedRetryDecision(
+            0, "job_preempted", RetryPlan(True, "retrying after preemption", 1)
+        ),
+    ).to_snapshot()
+    restored = RetryState.from_snapshot(spec, old_state)
+    assert not restored.persisted_plan(0).retry
+    assert restored.infra_used == 1
+    with pytest.raises(RuntimeError, match="lacks exact persisted retry authorization"):
+        require_retry_authorization_from_raw(spec, {state._RETRY_STATE_KEY: old_state}, 1)

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/tests/test_runmgmt.py

@@ -2476,7 +2476,10 @@ def test_attach_cache_fallback_restores_exact_shape_and_drops_managed_cache(monk
     assert retry["cache_retry_shape"] == ["modal", "H100", 2]
 
 
-def test_nonretryable_attached_failure_tears_down_without_resubmitting(monkeypatch, tmp_path):
+@pytest.mark.parametrize("failure", ["job_failed", "job_preempted"])
+def test_nonretryable_attached_failure_tears_down_without_resubmitting(
+    monkeypatch, tmp_path, failure
+):
     import io
 
     from flash.core.spec import GpuSpec, JobSpec
@@ -2504,7 +2507,7 @@ def test_nonretryable_attached_failure_tears_down_without_resubmitting(monkeypat
 
     class Provider:
         def poll_attempt(self, *_args, **_kwargs):
-            return PollResult(False, failure="job_failed", detail="worker assertion")
+            return PollResult(False, failure=failure, detail="worker stopped")
 
         def cancel(self, _handle):
             teardown.append("cancel")
@@ -2513,6 +2516,7 @@ def test_nonretryable_attached_failure_tears_down_without_resubmitting(monkeypat
             teardown.append("destroy")
 
     monkeypatch.setattr(providers, "get_provider", lambda _name: Provider())
+    monkeypatch.setattr(runner_recovery, "_completed_worker_metrics", lambda *_a, **_k: None)
     monkeypatch.setattr(runner_recovery, "_gc_run_sandboxes", lambda _spec: None)
     monkeypatch.setattr(
         runner_lifecycle,

{freesolo_flash_dev-1.4.6 → freesolo_flash_dev-1.4.7}/tests/test_server_auth_singleflight.py

@@ -40,10 +40,12 @@ def _clear_verify_state():
     with auth._verify_cache_lock:
         auth._verify_cache.clear()
         auth._verify_inflight.clear()
+        auth._verify_last_good.clear()
     yield
     with auth._verify_cache_lock:
         auth._verify_cache.clear()
         auth._verify_inflight.clear()
+        auth._verify_last_good.clear()
 
 
 def _http_error(code: int) -> urllib.error.HTTPError:
@@ -289,3 +291,88 @@ def test_local_disabled_key_stays_rejected_after_upstream_allow(monkeypatch, tmp
         )
 
     assert auth.authenticate("Bearer fslo-disabled") is None
+
+
+def _scripted_urlopen(monkeypatch, outcomes: list[object]) -> list[object]:
+    def fake_urlopen(req, timeout=None):
+        outcome = outcomes.pop(0)
+        if isinstance(outcome, BaseException):
+            raise outcome
+        return outcome
+
+    monkeypatch.setattr(auth.urllib.request, "urlopen", fake_urlopen)
+    return outcomes
+
+
+@pytest.mark.parametrize(
+    "outage",
+    [urllib.error.URLError("connection reset"), _http_error(503)],
+    ids=["transport", "server-error"],
+)
+def test_recently_verified_key_survives_identity_service_outage(monkeypatch, outage) -> None:
+    outcomes = _scripted_urlopen(monkeypatch, [_Response(), outage, _Response()])
+
+    assert auth._freesolo_verify("rollout-key") == _IDENTITY
+    # the outage is bridged by the identity verified moments ago, and nothing is negatively cached.
+    assert auth._freesolo_verify("rollout-key") == _IDENTITY
+    assert auth._verify_key_digest("rollout-key") not in auth._verify_cache
+    assert auth._verify_key_digest("rollout-key") not in auth._verify_inflight
+    # once the service answers again every request is verified live as before.
+    assert auth._freesolo_verify("rollout-key") == _IDENTITY
+    assert outcomes == []
+
+
+def test_outage_after_the_grace_window_still_fails(monkeypatch) -> None:
+    clock = [1000.0]
+    monkeypatch.setattr(auth.time, "monotonic", lambda: clock[0])
+    _scripted_urlopen(monkeypatch, [_Response(), _http_error(503)])
+
+    assert auth._freesolo_verify("old-key") == _IDENTITY
+    clock[0] += auth._VERIFY_STALE_GRACE_S + 1
+    with pytest.raises(auth.AuthVerificationUnavailable):
+        auth._freesolo_verify("old-key")
+
+
+def test_definitive_rejection_is_never_bridged_by_an_earlier_success(monkeypatch) -> None:
+    clock = [1000.0]
+    monkeypatch.setattr(auth.time, "monotonic", lambda: clock[0])
+    _scripted_urlopen(monkeypatch, [_Response(), _http_error(401), _http_error(503)])
+
+    assert auth._freesolo_verify("revoked-key") == _IDENTITY
+    assert auth._freesolo_verify("revoked-key") is None
+    assert auth._verify_key_digest("revoked-key") not in auth._verify_last_good
+    # after the negative cache expires, an outage must not resurrect the revoked key.
+    clock[0] += auth._VERIFY_CACHE_NEG_TTL_S + 1
+    with pytest.raises(auth.AuthVerificationUnavailable):
+        auth._freesolo_verify("revoked-key")
+
+
+def test_concurrent_waiters_share_the_bridged_identity(monkeypatch) -> None:
+    worker_count = 8
+    start = threading.Barrier(worker_count)
+
+    def fake_urlopen(req, timeout=None):
+        time.sleep(0.1)
+        raise urllib.error.URLError("connection reset")
+
+    with auth._verify_cache_lock:
+        auth._remember_good_locked(auth._verify_key_digest("shared"), _IDENTITY, time.monotonic())
+    monkeypatch.setattr(auth.urllib.request, "urlopen", fake_urlopen)
+
+    def verify() -> dict[str, object] | None:
+        start.wait()
+        return auth._freesolo_verify("shared")
+
+    with ThreadPoolExecutor(max_workers=worker_count) as pool:
+        results = list(pool.map(lambda _: verify(), range(worker_count)))
+    assert results == [_IDENTITY] * worker_count
+
+
+def test_last_good_identities_are_bounded_and_keep_no_token(monkeypatch) -> None:
+    monkeypatch.setattr(auth, "_VERIFY_CACHE_MAX", 3)
+    monkeypatch.setattr(auth.urllib.request, "urlopen", lambda req, timeout=None: _Response())
+
+    for index in range(5):
+        assert auth._freesolo_verify(f"key-{index}") == _IDENTITY
+    assert len(auth._verify_last_good) == 3
+    assert b"key-4" not in repr(auth._verify_last_good).encode()