freesolo-flash-dev 1.2.28__tar.gz → 1.2.36__tar.gz

freesolo_flash_dev-1.2.36/CHANGELOG.md

@@ -0,0 +1,113 @@
+# Changelog
+
+Notable changes to `freesolo-flash`. The dev-channel package `freesolo-flash-dev` ships in
+lockstep at the same version.
+
+Releases happen by merging a version bump from `dev` into `main`, which publishes to PyPI
+(`.github/workflows/publish.yml`). Versions are `MAJOR.MINOR.PATCH`.
+
+This file starts at 1.1.35. Earlier releases are not reconstructed here; use
+`git log` for anything before that.
+
+## Unreleased
+
+### Added
+
+- A `flash-cli` console script, the same entry point as `flash` under a name nothing else
+  claims. The `server` and `dev` extras install `runpod-flash`, which declares its own `flash`
+  script; whichever distribution is installed last wins, so on a control-plane host
+  `pip install 'freesolo-flash[server]'` could leave `flash` running RunPod's CLI, which exits 0
+  and does nothing. `flash` is unchanged and remains correct on a base (client-only) install.
+
+### Changed
+
+- CI runs the offline test suite on both supported interpreters (3.11 and 3.12) rather than
+  3.11 alone, and sets `HF_HUB_OFFLINE` / `TRANSFORMERS_OFFLINE` so a test that reaches the
+  network fails deterministically instead of depending on runner connectivity.
+- Every GitHub Action is pinned to a commit SHA rather than a mutable tag.
+- `mypy` runs in CI as an advisory (non-blocking) job.
+
+### Fixed
+
+- `flash models deploy` now warns before it moves the shared `<run-id>` model id onto a
+  different checkpoint. Every checkpoint of a run is served under that one id, so deploying
+  `<run-id>/step-50` while `step-100` was live silently changed what `<run-id>` served for
+  everyone using it — which made "deploy a second checkpoint to compare" a destructive
+  operation on the first, and read as a serving regression rather than the deploy that caused
+  it. The warning names both checkpoints and points at `flash models chat <run-id>/step-N`,
+  which addresses a checkpoint directly and does not depend on where the id points. It is
+  advisory: the deploy still proceeds, the pre-deploy read is abandoned outright once it exceeds
+  a few seconds so no shape of slow plane can hold a deploy behind it, and a control plane that
+  cannot answer (or answers with a checkpoint step this client cannot read) does not fail the
+  command. A deployment whose activation outcome was never settled still warns, but without
+  naming a checkpoint: that record's step describes the incoming attempt rather than what the id
+  currently serves, and the authoritative target is server-side only, so the warning says the
+  live checkpoint cannot be determined and points at `flash models deployments`.
+
+- A response deadline did not actually bound a response that arrived slowly. The deadline is
+  checked between reads, but each read asked for a fixed number of bytes and blocked until all of
+  them arrived, so a peer trickling a short body kept one read inside the socket timeout
+  indefinitely and the check never ran: a 2s deadline took 12s on a 42-byte body, and 61s when
+  the socket timeout was the client default. Reading whatever has already arrived lets the check
+  run, which bounds every stall shape. This affected `flash env list`, whose deadline exists to
+  stop a slow environment hub hanging the command, as well as the new pre-deploy read.
+
+- A response deadline could still be overrun by a read that began near its end. The socket timeout
+  is installed once, when the request opens, and time spent connecting and waiting for headers is
+  charged to the deadline but not to that timeout, so a peer that delayed its headers and then
+  stalled mid-body held the call open past the deadline: 3.5s against a 2s budget. Each read now
+  re-caps the socket to the remaining budget, so the two bounds agree instead of stacking. On a
+  body still arriving this surfaces as a timeout rather than the "stalled" message; both bound the
+  call, and which one wins is a race not worth depending on.
+
+- Commands printed for the operator to run (the resume/cancel hand-off after `flash train`,
+  usage strings, `next:` hints) now name the executable actually invoked rather than always
+  `flash`. On a host where `runpod-flash` owns the `flash` script, the printed
+  `flash runs cancel <run-id>` exited 0 without cancelling, leaving a run billing while the
+  operator believed they had stopped it. Reached through the `python -m flash.cli` escape hatch,
+  the printed command now names the interpreter that is actually running, rather than a bare
+  `python` that is absent on a python3-only host and may resolve to a different environment
+  inside a virtualenv.
+
+- `flash env setup` scaffolded hosted-only environment instructions on every plane. A
+  self-hosted plane cannot publish to Freesolo's managed environment hub, so the generated
+  configs left `[environment] id` empty (which fails validation) beside a `flash env push`
+  command that could not work. Against a self-hosted plane the scaffold now emits the
+  `github:owner/repo@ref:path` id form, which such a plane resolves directly. Output against
+  the managed plane is unchanged.
+
+- `flash-server` reported missing operator configuration as an unhandled ASGI startup
+  exception, so the actionable message arrived under ~20 frames of starlette/contextlib. The
+  refusing half of the preflight now runs before uvicorn starts, and `flash-server` prints
+  `error: ...` and exits 3 (uvicorn's `STARTUP_FAILURE`, which this path already used, so
+  supervision keying on it is unaffected). The advisory warnings still come from the lifespan
+  alone, so a booted plane logs them once.
+- `flash env test` blamed the reward function for a gold answer that scored zero, when the
+  gold completion is equally likely to be at fault: with no `sft_completion` hook the replayed
+  answer is the dataset row's raw `output`, so a dataset whose `output` is a bare value fails a
+  grader that (correctly) requires a wrapper. The warning now names both candidates and prints
+  the exact text it scored, verbatim rather than through the whitespace-collapsing, 200-character
+  preview that hid the formatting faults it exists to reveal. The scaffolded `environment.py`
+  documents that `output` is also the gold assistant turn SFT trains on, and that per-row scorer
+  state belongs in `metadata`.
+- `flash env test` labelled the replayed dataset turns as the text that scored zero, but a
+  multi-turn environment whose `step_episode` returns a `final_response_text` has that override
+  replace the episode's response before scoring, so the diagnostic named text the grader never
+  saw. It now reports what the scorer actually received, captured at the scoring call.
+- An environment that overrode its answer to the empty string had `flash env test` report the
+  replayed turns instead: the captured text was tested for truth, so a graded `""` was
+  indistinguishable from an episode that never reached the scorer. The override propagates on
+  `is not None`, so `""` really is what the grader saw -- and an empty answer explains a zero
+  reward on its own, making it the reading a reader most needs named rather than hidden.
+- A crashed scorer went unreported on an `echo` episode -- a row with no gold answer to replay,
+  which is also the case that leaves the GRPO gate with nothing to count. `flash env test` could
+  exit `overall: PASS` with every reward coming from a scorer that never ran. The scorer's error is
+  now surfaced for any policy.
+- A scorer that crashed behind the SDK's guard reached `flash env test` as a bare `0.000000`,
+  indistinguishable from a judged-wrong answer, because the adapter kept only
+  `RewardResult.score`. The new `FreesoloEnvironment.reward_with_error` returns the reward and the
+  discarded error from a single scoring call, so a missing runtime dependency names itself without
+  re-scoring the episode -- which would bill a paid judge twice and, for a non-deterministic
+  scorer, could report an error that did not produce the printed reward.
+- Two tests resolved Hugging Face model metadata over the network and passed only on a
+  connected runner. They now stub the resolver like their siblings already did.

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: freesolo-flash-dev
-Version: 1.2.28
+Version: 1.2.36
 Summary: Flash — managed LoRA post-training (SFT/GRPO/OPD) for Freesolo environments, driven by the `flash` CLI
 Project-URL: Homepage, https://github.com/freesolo-co/flash
 Project-URL: Repository, https://github.com/freesolo-co/flash

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/flash/cli/__init__.py

@@ -738,11 +738,11 @@ def _add_deployment_commands(models_sub: argparse._SubParsersAction) -> None:
         "--wait",
         nargs="?",
         type=_wait_seconds,
-        const=1800.0,
+        const=2400.0,
         default=None,
         metavar="SECONDS",
         help=(
-            "block until the requested revision is ready or failed (default timeout 1800s). "
+            "block until the requested revision is ready or failed (default timeout 2400s). "
             "without it, deploy returns while the revision is still queued"
         ),
     )

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/flash/cli/commands/deploy.py

@@ -11,6 +11,7 @@ from __future__ import annotations
 
 import json
 import sys
+import threading
 import time
 
 from flash.cli.ui import render
@@ -41,6 +42,10 @@ _DEPLOYMENT_BUSY_STATES = frozenset({"queued", "smoke_testing", "reconciling"})
 # plane introduces are both non-busy and non-servable, so `--wait` must fail closed on them
 # rather than let `deploy --wait && evaluate` proceed against nothing.
 _DEPLOYMENT_READY_STATES = frozenset({"ready", "deployed"})
+# an undeploy whose backend cleanup failed (flash/runner/supervise/transitions.py). local serving
+# authority is revoked, but the alias may still resolve to the old target -- neither servable nor
+# safely assumed idle.
+_REVOCATION_FAILED_STATE = "revocation_failed"
 _DEPLOY_POLL_SECONDS = 5.0
 # `--wait 0` still owes the caller its one read, and a read needs a positive timeout. keep that
 # bound short enough that "check once, do not block" stays true against a stalled plane: a longer
@@ -57,6 +62,14 @@ _DEPLOY_FINAL_READ_FRACTION = 0.9
 # an auth or authorization rejection answers the same way every time; polling through it just
 # spends the whole timeout to arrive at the identical error.
 _PERMANENT_POLL_STATUSES = frozenset({401, 403})
+# the pre-deploy alias read is advisory, so it must not spend the client's default 60s budget
+# deciding whether to print a warning: a stalled read would delay every real deploy behind a
+# line nobody asked for. short enough to stay unnoticed, long enough for a healthy plane.
+# it is spent three ways, because the client's own bounds are per-phase: as a socket timeout it
+# bounds one socket operation, and as a body deadline it bounds the body but is only consulted
+# once headers are parsed. a peer trickling headers just inside the socket timeout escapes both.
+# `_read_within` bounds the total in wall-clock time, which is the one that actually holds.
+_ALIAS_WARNING_READ_SECONDS = 5.0
 
 
 def _await_deployment(client, run_id: str, deployment: dict, timeout: float) -> dict:
@@ -189,6 +202,196 @@ def _rollback_record(client, run_id: str, timeout: float) -> dict | None:
     return None
 
 
+def _served_step_label(step: int | None) -> str:
+    """Name a deployment's checkpoint the way the user addressed it: `step-N`, or `final`."""
+    return "final" if step is None else f"step-{step}"
+
+
+def _read_within(budget: float, read):
+    """Run an advisory read, giving up on it entirely once ``budget`` seconds have passed.
+
+    The client's two bounds are both per-phase, not total: `timeout` restarts on every socket
+    operation and the body deadline is only checked once `urlopen` has finished parsing headers.
+    A peer that trickles the status line or headers just inside the socket timeout therefore
+    stalls for timeout x however many pauses it takes -- 12s against a 2s budget, measured -- and
+    the whole point of the bound is that the deploy the user actually asked for is waiting behind
+    it. Only wall-clock time outside the call bounds every phase at once.
+
+    A daemon thread rather than a cancellation: a blocked socket read cannot be interrupted from
+    outside, so the read is abandoned rather than stopped. That is sound only because this result
+    is discardable -- no warning is the documented outcome for a plane that cannot answer, so an
+    overrunning read degrades to silence. Daemon, so a stuck thread cannot keep the interpreter
+    alive after the deploy finishes.
+    """
+    result: list = []
+    worker = threading.Thread(
+        target=lambda: result.append(_read_or_none(read)),
+        daemon=True,
+    )
+    worker.start()
+    worker.join(budget)
+    return result[0] if result else None
+
+
+def _read_or_none(read):
+    """The advisory read's failure contract: a plane that cannot answer produces no warning.
+
+    Deliberately broader than `ApiError`/`ClientError`. The client translates urllib's HTTPError
+    and URLError, and documents that anything else propagates -- so a connection reset or a
+    truncated body mid-read surfaces as the bare `ConnectionResetError` / `IncompleteRead` that
+    urllib raised. Those are exactly the "plane cannot answer" case this exists to absorb, and on
+    the worker thread an escaping one reaches the thread hook and prints a traceback across a
+    deploy that is otherwise proceeding normally: a warning nobody asked for turning into noise
+    on a command that worked.
+    """
+    try:
+        return read()
+    except Exception:
+        # the deploy itself is the authority on whether it can proceed. failing it here -- or
+        # printing a traceback beside it -- would turn a warning nobody asked for into a fault in
+        # the command it decorates. BaseException still propagates, so KeyboardInterrupt is
+        # unaffected.
+        return None
+
+
+def _alias_move_warning(client, base_run_id: str, requested_step: int | None) -> str | None:
+    """Warn when deploying moves the shared `<run_id>` alias off a different checkpoint.
+
+    `deploy` registers under the bare run id whatever step it is given, so deploying a second
+    checkpoint of the same run does not stand up a second endpoint -- it repoints the one alias,
+    and everyone chatting bare `<run_id>` silently changes model. That makes "deploy another
+    checkpoint to compare" a destructive operation on the first, which reads as a serving fault
+    rather than the deploy that caused it.
+
+    Best-effort by construction: the read is advisory, so a plane that cannot answer must not
+    stop the deploy. Returns None when nothing is being displaced -- no deployment, the same
+    checkpoint again, or an unreadable current record.
+    """
+    current = _read_within(
+        _ALIAS_WARNING_READ_SECONDS,
+        # not `deployment_for`: its step filter hides exactly the record this asks about, a
+        # DIFFERENT checkpoint holding the alias.
+        lambda: client.deployed_checkpoint(
+            base_run_id,
+            timeout=_ALIAS_WARNING_READ_SECONDS,
+            body_deadline=_ALIAS_WARNING_READ_SECONDS,
+        ),
+    )
+    if current is None:
+        return None
+    # a `reconciling` record whose activation outcome was never recorded may ALREADY hold the
+    # alias. the plane permits replacing exactly that record rather than rejecting it as busy,
+    # and resolves the authoritative target through `_activation_predecessor` when it does
+    # (flash/server/routes/serving.py). reading it as "not serving" suppressed the warning in
+    # the very case the alias is most likely to move out from under someone.
+    unknown_activation = current.get("activation_outcome_unknown") is True
+    # `revocation_failed` is an undeploy whose BACKEND cleanup failed: local authority is revoked
+    # but the alias may still resolve to the old target, and `_validate_deploy_request` rejects
+    # only the busy set, so a deploy proceeds from here. reading it as "not serving" suppressed
+    # the warning in the case where cleanup already left the alias ambiguous.
+    revocation_failed = str(current.get("state") or "") == _REVOCATION_FAILED_STATE
+    ambiguous = unknown_activation or revocation_failed
+    if str(current.get("state") or "") not in _DEPLOYMENT_READY_STATES and not ambiguous:
+        # nothing is being served off the alias yet, so nothing is lost by moving it.
+        return None
+    cli = _commands().CLI_NAME
+    # `step-N` is not universally available: `ApiClient.chat` gates a step target behind
+    # `_require_chat_step_selector`, which refuses outright on a plane that does not advertise
+    # `chat_step_selector_v1`. the capability cannot be read here without spending a /v1/health on
+    # an advisory warning, and `_chat_step_selector_available` starts False, so a negative reading
+    # cannot tell "unsupported" from "not checked yet". state the condition instead of asserting
+    # the command works, and name the selector that works on every plane.
+    tail = (
+        f"so every client using bare `{base_run_id}` changes model. address a specific "
+        f"checkpoint with `{cli} models chat {base_run_id}/step-N` to compare them, or by "
+        "immutable adapter revision if this control plane does not support step selectors."
+    )
+    if revocation_failed:
+        # same reason as the unknown-activation arm: this record's `checkpoint_step` describes the
+        # deployment whose revocation failed, and whether the backend still answers on the alias
+        # is exactly what could not be determined. hedge rather than name a checkpoint.
+        return (
+            f"{base_run_id} has a deployment whose revocation did not complete, so whether it "
+            f"still serves off that shared model id cannot be determined from here; deploying "
+            f"{_served_step_label(requested_step)} may move it, "
+            f"{tail} run `{cli} models deployments` to see the current state first."
+        )
+    if unknown_activation:
+        # this record's `checkpoint_step` names the INCOMING attempt, not what the alias holds:
+        # the plane resolves the live target from `adapter_alias_target` / `previous_deployment`,
+        # and both are server-side (`public_deployment` strips the predecessor). naming a
+        # checkpoint here would report the wrong one -- worse than naming none -- so say only what
+        # this side can actually stand behind.
+        return (
+            f"{base_run_id} has a deployment whose activation never settled, so the checkpoint "
+            f"it currently serves cannot be determined from here; deploying "
+            f"{_served_step_label(requested_step)} may move that shared model id, "
+            f"{tail} run `{cli} models deployments` to see the current state first."
+        )
+    raw_step = current.get("checkpoint_step")
+    # `int()` does not just reject what it cannot read -- it silently COERCES. `true` becomes 1,
+    # `false` becomes 0, and `1.5` truncates to 1, none of them raising. a step this client cannot
+    # read as an exact whole number is an unreadable record, not checkpoint 1: reading it as a
+    # number either suppresses the warning (deploying step-1 against a `true`) or names a
+    # checkpoint that was never live. bool is checked first because it is a subclass of int.
+    if isinstance(raw_step, bool):
+        return None
+    if isinstance(raw_step, float) and not raw_step.is_integer():
+        return None
+    try:
+        served_step = None if raw_step is None else int(raw_step)
+    except (TypeError, ValueError, OverflowError):
+        # a proxy or older plane can answer 2xx with a step this client cannot read. that is an
+        # unreadable current record like any other, NOT a reason to fail the deploy: leaving the
+        # conversion unguarded let an advisory read traceback out of the command it decorates.
+        # all three arms are reachable from one JSON body: `json.loads` accepts the non-standard
+        # `Infinity`/`NaN` literals, and int() answers those with OverflowError and ValueError
+        # respectively -- so catching only the obvious two still crashed the deploy.
+        return None
+    if served_step == requested_step:
+        return None
+    reach_displaced = ""
+    # the record being displaced carries the one identifier that survives the alias move, so name
+    # it whenever it is there. `step-N` is not a substitute: it is rejected outright on a plane
+    # without `chat_step_selector_v1`, and after this deploy the listing shows only the NEW record
+    # -- `public_deployment` strips `previous_deployment` (flash/serve/urls.py) -- so a revision
+    # not printed here is not recoverable from anywhere afterwards.
+    revision = str(current.get("adapter_revision") or "").strip()
+    if served_step is None:
+        # the displaced checkpoint is the FINAL adapter, and `_parse_chat_target` accepts only a
+        # bare run id, `RUN/step-N`, or a full immutable revision -- there is no `/final`
+        # selector. once the alias moves, `step-N` addresses the incoming checkpoint and the bare
+        # id is the thing that just changed, so the generic advice cannot reach the final adapter
+        # at all. its immutable revision can, and the record being displaced carries one.
+        reach_displaced = (
+            f" the displaced final adapter has no `/final` selector; reach it by its immutable "
+            f"revision `{revision}`."
+            if revision
+            # without a revision on the record there is nothing to hand the user, and deferring
+            # them to a later command would be worse than saying so: `client.deploy` fires as soon
+            # as this prints, and the deployments listing then shows only the NEW record --
+            # `public_deployment` strips `previous_deployment` (flash/serve/urls.py), so the
+            # displaced revision is not in it. promising a selector that the next command cannot
+            # produce is the one outcome worse than admitting there is none.
+            else (
+                " the displaced final adapter has no `/final` selector and this plane did not "
+                "report its immutable revision, so it will not be addressable after this deploy; "
+                "undeploy and redeploy it to serve it again."
+            )
+        )
+    elif revision:
+        # a numbered checkpoint keeps its `step-N` selector, but only on a plane that advertises
+        # `chat_step_selector_v1`; where it does not, the qualified advice above leaves the user
+        # holding no identifier at all. this is that identifier, and this is the last moment it
+        # can be printed.
+        reach_displaced = f" the displaced checkpoint's immutable revision is `{revision}`."
+    return (
+        f"{base_run_id} currently serves {_served_step_label(served_step)}; deploying "
+        f"{_served_step_label(requested_step)} moves that shared model id onto the new "
+        f"checkpoint, {tail}{reach_displaced}"
+    )
+
+
 def _deployment_attempt_failed(requested: dict, final: dict) -> bool:
     """True when the requested revision is not the one now served.
 
@@ -229,8 +432,18 @@ def cmd_deploy(args) -> int:
             file=sys.stderr,
         )
         return 1
-    base_run_id, _step = parsed
+    base_run_id, step = parsed
     client = _commands().client_from_config()
+    # before the POST: after it the alias has already moved, and a warning about a checkpoint that
+    # is no longer served reads as history rather than a decision the reader still has. a dry run
+    # registers nothing, so it displaces nothing and gets no warning.
+    if not args.dry_run:
+        alias_warning = _alias_move_warning(client, base_run_id, step)
+        if alias_warning:
+            print(
+                render.warn(alias_warning) if render.styled() else f"warning: {alias_warning}",
+                file=sys.stderr,
+            )
     dep = client.deploy(args.run_id, dry_run=args.dry_run)
     wait_seconds = getattr(args, "wait", None)
     # a dry run creates no deployment to poll for, so --wait has nothing to observe. test against

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/flash/cli/commands/env/episode.py

@@ -13,9 +13,11 @@ from __future__ import annotations
 
 import contextlib
 import inspect
+import sys
 
 from flash.cli.commands.env.test_evaluations import _evaluation_example
-from flash.envs.evaluations import EvalCase, EvalResult
+from flash.cli.ui import render
+from flash.envs.evaluations import EvalCase, EvalResult, _episode_grading_enabled
 
 
 def _eval_module():
@@ -34,7 +36,7 @@ def _grades_episodes(suite) -> bool:
     Keying off the environment scored the wrong turn there and turned a well-formed first action
     into an error, so the suite has to say so itself.
     """
-    return bool(getattr(suite, "grades_episodes", False))
+    return _episode_grading_enabled(suite)
 
 
 def _drive_episode(client, target: str, environment, case: EvalCase, args) -> dict | str:
@@ -123,7 +125,14 @@ def _effective_turn_cap(environment, state: dict) -> int:
 
 
 def _score_episode_case(
-    suite, case: EvalCase, case_id: str, state: dict, *, thinking: bool = False
+    suite,
+    case: EvalCase,
+    case_id: str,
+    state: dict,
+    *,
+    scorer,
+    state_style: str | None,
+    thinking: bool = False,
 ) -> EvalResult:
     """Grade a finished episode through the suite's own scorer.
 
@@ -134,18 +143,19 @@ def _score_episode_case(
     `reward(completion, example, state)`, which is what reaches the SDK's `score_episodes`.
 
     `EvalSuite.score(case, response)` is the published two-argument contract, so state is offered
-    only to a suite that accepts it, detected by signature rather than by try/except: a TypeError
-    raised INSIDE a scorer that does accept state would otherwise be silently retried as a
-    two-argument call and graded on the wrong text.
+    only to a suite that accepts it. The caller resolves one scorer and signature style for the
+    whole suite rather than retrying on TypeError: an error raised INSIDE a state-aware scorer must
+    not be retried as a two-argument call and graded on the wrong text.
     """
     response = state.get("response_text")
     if not isinstance(response, str):
         turns = state.get("turns") or []
         response = str(turns[-1]) if turns else ""
     eval_module = _eval_module()
-    style = _state_argument(getattr(suite, "score", None))
-    if style is None:
-        return eval_module._score_case(suite, case, case_id, response, thinking=thinking)
+    if state_style is None:
+        return eval_module._score_case(
+            suite, case, case_id, response, thinking=thinking, scorer=scorer
+        )
     return eval_module._score_case(
         suite,
         case,
@@ -153,7 +163,8 @@ def _score_episode_case(
         response,
         thinking=thinking,
         state=state,
-        state_keyword=style == "keyword",
+        state_keyword=state_style == "keyword",
+        scorer=scorer,
     )
 
 
@@ -173,27 +184,45 @@ def _state_argument(score) -> str | None:
     except (TypeError, ValueError):
         # builtins and C callables expose no signature; treat them as the published contract.
         return None
-    named_state = next((p for p in parameters if p.name == "state"), None)
-    if named_state is not None and named_state.kind in (
-        inspect.Parameter.KEYWORD_ONLY,
-        inspect.Parameter.POSITIONAL_OR_KEYWORD,
-    ):
-        # By keyword even where positional would also bind: it cannot land on the wrong parameter.
-        return "keyword"
-    # Checked before `*args`, since a scorer with both can take `state=` but one with only
-    # `*args` cannot.
-    if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in parameters):
-        return "keyword"
-    if any(p.kind is inspect.Parameter.VAR_POSITIONAL for p in parameters):
-        return "positional"
     positional = [
-        p
-        for p in parameters
-        if p.kind in (inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)
+        parameter
+        for parameter in parameters
+        if parameter.kind
+        in (inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)
     ]
-    # (case, response) is the published contract; a third positional is the episode state. This
-    # also catches a positional-only `state`, which the named check above deliberately skips.
-    return "positional" if len(positional) >= 3 else None
+    named_state = next((parameter for parameter in parameters if parameter.name == "state"), None)
+    if named_state is not None:
+        if named_state.kind is inspect.Parameter.POSITIONAL_ONLY:
+            return "positional" if positional.index(named_state) == 2 else None
+        if named_state.kind in (
+            inspect.Parameter.KEYWORD_ONLY,
+            inspect.Parameter.POSITIONAL_OR_KEYWORD,
+        ):
+            # by keyword even where positional would also bind: it cannot land on the wrong parameter.
+            return "keyword"
+    # checked before `*args`, since a scorer with both can take `state=` but one with only
+    # `*args` cannot.
+    if any(parameter.kind is inspect.Parameter.VAR_KEYWORD for parameter in parameters):
+        return "keyword"
+    if any(parameter.kind is inspect.Parameter.VAR_POSITIONAL for parameter in parameters):
+        # scorer(case, response, state) reaches *args when the signature has no fixed third slot.
+        # zero, one, or two regular positional parameters can all bind that call.
+        return "positional" if len(positional) <= 2 else None
+    return None
+
+
+def _warn_if_episode_state_is_hidden(suite, state_style: str | None) -> None:
+    """Warn once when full-episode work feeds a scorer that cannot see the transcript."""
+    if state_style is not None:
+        return
+    message = (
+        f"suite {getattr(suite, 'name', 'evaluation')!r} sets grades_episodes = True; "
+        "each episode will still be played out with one generation per turn, but the scorer "
+        "will receive only the episode's final response text, not the transcript. Add a third "
+        "`state` argument to "
+        "`score(case, response, state)` to grade the transcript."
+    )
+    print(render.warn(message) if render.styled() else f"warning: {message}", file=sys.stderr)
 
 
 def _run_episode_cases(
@@ -220,6 +249,9 @@ def _run_episode_cases(
             )
             for case_id in case_ids
         )
+    scorer = getattr(suite, "score", None)
+    state_style = _state_argument(scorer)
+    _warn_if_episode_state_is_hidden(suite, state_style)
     results = []
     # The adapter defaults to `thinking = False` (flash/envs/adapter.py), and with it off
     # `_scored_turn_text` returns the turn unstripped -- so `state["response_text"]` keeps its
@@ -237,7 +269,17 @@ def _run_episode_cases(
             if isinstance(outcome, str):
                 results.append(eval_module._generation_error(case_id, outcome))
                 continue
-            results.append(_score_episode_case(suite, case, case_id, outcome, thinking=thinking))
+            results.append(
+                _score_episode_case(
+                    suite,
+                    case,
+                    case_id,
+                    outcome,
+                    scorer=scorer,
+                    state_style=state_style,
+                    thinking=thinking,
+                )
+            )
     return tuple(results)
 
 

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/flash/cli/commands/env/eval.py

@@ -162,6 +162,7 @@ def _score_case(
     thinking: bool = False,
     state=None,
     state_keyword: bool = False,
+    scorer=None,
 ) -> EvalResult:
     """Grade one response on the caller's thread.
 
@@ -172,7 +173,8 @@ def _score_case(
     `episode._score_episode_case`); single-turn scoring keeps the two-argument contract.
     `state_keyword` says which way that suite's signature can take it -- `**kwargs` and
     keyword-only scorers reject a third positional, `*args` scorers reject the keyword -- so the
-    caller decides, having read the signature.
+    caller decides, having read the signature. Episode scoring also passes the exact callable it
+    inspected, so a descriptor cannot change shape between capability detection and invocation.
 
     The scorer grades the stripped answer; the RESULT records the raw generation. Stripping for the
     scorer must not shorten what is recorded, on the error path least of all -- a failed case is
@@ -182,12 +184,13 @@ def _score_case(
     try:
         scored_text = _scored_response(response, thinking=thinking)
         recorded_response = getattr(scored_text, "raw", response)
+        score = suite.score if scorer is None else scorer
         if state is None:
-            scored = suite.score(case, scored_text)
+            scored = score(case, scored_text)
         elif state_keyword:
-            scored = suite.score(case, scored_text, state=state)
+            scored = score(case, scored_text, state=state)
         else:
-            scored = suite.score(case, scored_text, state)
+            scored = score(case, scored_text, state)
         return normalize_eval_result(case, recorded_response, scored, case_id=case_id)
     except (Exception, SystemExit) as exc:
         return EvalResult(

{freesolo_flash_dev-1.2.28 → freesolo_flash_dev-1.2.36}/flash/cli/commands/env/setup.py

@@ -138,12 +138,14 @@ Given the opening prompt, does the model emit the single integer `step_episode`
 to advance the game? That is a real held-out check of the action format the episode
 depends on, and it is the honest scope of a single-shot evaluation.
 
-To grade a finished transcript instead, set `grades_episodes = True` on the suite.
-`env eval` then plays each case out against the deployed model -- generating, stepping
-the environment, and repeating to `max_episode_turns` -- and scores the resulting
-transcript. Such a suite needs cases the environment can actually advance (the cases
-below carry no `output`, so `step_episode` has no secret to compare against), and each
-case then costs one generation per turn rather than one in total.
+To grade a finished transcript instead, set `grades_episodes = True` on the suite
+and define `score(self, case, response, state)`. `env eval` then plays each case out
+against the deployed model -- generating, stepping the environment, and repeating to
+`max_episode_turns` -- and passes the resulting transcript in `state`; without that
+argument the scorer receives only the episode's final response text. Such a suite
+needs cases the environment can actually advance (the cases below carry no `output`,
+so `step_episode` has no secret to compare against), and each case then costs one
+generation per turn rather than one in total.
 
 Do NOT grade these by calling `environment.reward(response, example)`: with no episode
 state that call scores an empty transcript, so an unrelated answer can score 1.0 and
@@ -289,7 +291,7 @@ class StarterMultiTurnEnv(EnvironmentMultiTurn):
         # Advance the world after one assistant action. Return done=True to end the
         # episode, or append an observation message and keep going.
         # `messages` already ends with this action: messages[-1]["content"] is
-        # `assistant_response`. If you rebuild state by replaying the transcript, replay
+        # `assistant_response`. if you rebuild state by replaying the transcript, replay
         # `messages[:-1]` and apply `assistant_response` once, or you apply it twice.
         try:
             guess = int(assistant_response.strip().split()[0])