prefect 3.9.1.dev8__py3-none-any.whl → 3.9.2.dev2__py3-none-any.whl

prefect/AGENTS.md

@@ -29,7 +29,7 @@ Runtime flow/task parameter values must come from `utilities.callables.get_call_
 - **Sync and async must stay in sync.** Both `flow_engine.py` and `task_engine.py` have sync and async paths. Any behavior change must be applied to both.
 - **Event and log WebSocket subscribers are parallel implementations.** `events/clients.py::PrefectEventSubscriber` and `logging/clients.py::PrefectLogsSubscriber` duplicate the same reconnect/backoff/dedup logic, including the `reconnect_on_clean_close` option (used by `events/subscribers.py::FlowRunSubscriber` so a clean disconnect doesn't hide a terminal event). Mirror any reconnection-behavior change across both.
 - **Flow run observers own their stream recovery.** `_internal/observers.py` observers create subscribers with `reconnection_attempts=0` and delegate polling fallback, reconnect backoff, and setup deadlines to `_internal/_observation_delivery.py::ObservationDelivery`. Subscribers close any socket left by a failed or cancelled `__aenter__`, since callers never exit a context that failed to enter.
-- **Use `whenever` compat helpers for all datetime conversions on Python 3.13+.** `prefect-client` depends on `whenever` only on Python 3.13+, so modules it ships must keep `whenever` imports behind `sys.version_info >= (3, 13)`; only server code that the client build strips may import it unconditionally (see `client/AGENTS.md`). `types/_datetime.py` provides `now()`, `_whenever_to_stdlib()`, `_whenever_zdt_from_py()`, and `_whenever_pdt_from_py()` to abstract over API differences across supported whenever releases. Version-specific code is guarded by feature-detection flags (`_WHENEVER_NEW_API` for ≥ 0.10.0, `_SYSTEM_TZ` for ≥ 0.11), not version checks. Never call `DateTime.now()` or `pendulum.now()` directly — use `now()` from `types/_datetime.py` instead. Avoid no-argument `datetime.datetime.now()` because it returns a tz-naive datetime that raises `TypeError` when compared to the tz-aware datetimes produced by `now()`. Explicitly timezone-aware stdlib calls such as `datetime.now(ZoneInfo("UTC"))` are allowed when needed to preserve timezone information. Never call `ZonedDateTime.from_py_datetime()`, `PlainDateTime.from_py_datetime()`, or `.py_datetime()` directly — use the helpers instead. Likewise avoid `.assume_system_tz()`/`.to_system_tz()` (deprecated in 0.11; the warning fails tests under `filterwarnings = error`) — use `in_local_tz()` — and direct `DateTimeDelta` imports (removed in 0.11) — use the `Interval` alias. Violations will silently break on whichever whenever version the helpers weren't written for.
+- **Use `whenever` compat helpers for all datetime conversions on Python 3.13+.** `prefect-client` depends on `whenever` only on Python 3.13+, so modules it ships must keep `whenever` imports behind `sys.version_info >= (3, 13)`; only code that the client build strips may import it unconditionally (see the repo-root `client/AGENTS.md`). `types/_datetime.py` provides `now()`, `_whenever_to_stdlib()`, `_whenever_zdt_from_py()`, and `_whenever_pdt_from_py()` to abstract over API differences across supported whenever releases. Version-specific code is guarded by feature-detection flags (`_WHENEVER_NEW_API` for ≥ 0.10.0, `_SYSTEM_TZ` for ≥ 0.11), not version checks. These flags are only probed on Python 3.13+ (`_WHENEVER_NEW_API` is `False` and `_SYSTEM_TZ` is undefined on 3.11/3.12), so a helper that server-only code calls on every Python must feature-detect on the `whenever` class itself. Never call `DateTime.now()` or `pendulum.now()` directly — use `now()` from `types/_datetime.py` instead. Avoid no-argument `datetime.datetime.now()` because it returns a tz-naive datetime that raises `TypeError` when compared to the tz-aware datetimes produced by `now()`. Explicitly timezone-aware stdlib calls such as `datetime.now(ZoneInfo("UTC"))` are allowed when needed to preserve timezone information. Never call `ZonedDateTime.from_py_datetime()`, `PlainDateTime.from_py_datetime()`, or `.py_datetime()` directly — use the helpers instead. Likewise avoid `.assume_system_tz()`/`.to_system_tz()` (deprecated in 0.11; the warning fails tests under `filterwarnings = error`) — use `in_local_tz()` — and direct `DateTimeDelta` imports (removed in 0.11) — use the `Interval` alias. Violations will silently break on whichever whenever version the helpers weren't written for.
 - **Call Pendulum-only members via `types/_datetime.py::_pendulum_call` on Python 3.11/3.12.** Prefect's `DateTime`/`Date`/`Duration` wrap Pendulum-only members (and some shared stdlib methods like `replace`) with user-facing deprecation warnings; `_pendulum_call` bypasses them so users only see warnings for their own calls. `pyproject.toml` ignores these warnings in pytest, so direct calls from Prefect code pass tests silently unless covered by the guards in `tests/test_datetime.py`.
 - **Flow and task engines advance state differently.** The flow engine makes blocking API calls to the server to propose and advance states. The task engine emits `prefect.task_run.*` events (delivered via WebSockets) but advances state locally through `set_state` calls with polling/backoff — do not assume the two engines work the same way.
 - **Flow state transitions go through the server.** The flow engine proposes states to the server, which accepts or rejects them via orchestration rules. The task engine, by contrast, manages state transitions locally via `set_state` and emits `prefect.task_run.*` events — it does not propose states to the server.

prefect/_build_info.py

@@ -1,5 +1,5 @@
 # Generated by versioningit
-__version__ = "3.9.1.dev8"
-__build_date__ = "2026-10-30 08:15:31.522670+00:00"
-__git_commit__ = "599697348089e83823cb4ef63ff219c53eefec22"
+__version__ = "3.9.2.dev2"
+__build_date__ = "2026-11-02 08:16:54.718332+00:00"
+__git_commit__ = "da9d9dc15f647f10135a8ff79ce93c4bdb2dbf6a"
 __dirty__ = False

prefect/cli/AGENTS.md

@@ -7,7 +7,7 @@ Prefect command-line interface, powered by **cyclopts**.
 - **Use `rich` for all output.** Console output via `rich.console`, tables via `rich.table`, progress bars via `rich.progress`. Always use `exit_with_error` for error exits.
 - **Support JSON output whenever possible.** Commands should have a `--json` flag or equivalent for machine-readable output. Any diagnostic or informational `print()` calls must be suppressed when JSON output is active — mixing human-readable text with the JSON payload breaks machine consumers.
 - **Print JSON and other verbatim user data with `markup=False, emoji=False`.** `console.print` otherwise parses `[...]` as Rich markup and `:name:` as emoji aliases, silently rewriting payloads (or raising `MarkupError` on stray closing tags). `print_json` and `Pretty` are not affected.
-- **Page through "all records" reads.** Client `read_*` list calls without `limit` return at most `PREFECT_SERVER_API_DEFAULT_LIMIT` (200) rows (server `LimitBody`), so `--all`/bulk commands must loop on `offset`. When deleting while paging, re-read from offset 0 each pass — advancing `offset` skips rows that shifted into earlier pages.
+- **Page through "all records" reads.** Client `read_*` list calls without `limit` return at most `PREFECT_SERVER_API_DEFAULT_LIMIT` (200) rows (server `LimitBody`) — or fewer when the client method has its own default `limit` (e.g. `read_global_concurrency_limits` defaults to 10) — so `--all`/bulk commands must loop on `offset`. When deleting while paging, re-read from offset 0 each pass — advancing `offset` skips rows that shifted into earlier pages.
 - **Excluded from `prefect-client`** — the entire `cli/` directory is stripped during the client package build.
 
 ## Structure

prefect/cli/artifact.py

@@ -77,7 +77,7 @@ async def list_artifacts(
     if output and output.lower() == "json":
         artifacts_json = [artifact.model_dump(mode="json") for artifact in artifacts]
         json_output = orjson.dumps(artifacts_json, option=orjson.OPT_INDENT_2).decode()
-        _cli.console.print(json_output, soft_wrap=True)
+        _cli.console.print(json_output, soft_wrap=True, markup=False, emoji=False)
     else:
         table = Table(
             title="Artifacts",
@@ -181,7 +181,7 @@ async def inspect(
             json_output = orjson.dumps(
                 artifacts_json, option=orjson.OPT_INDENT_2
             ).decode()
-            _cli.console.print(json_output, soft_wrap=True)
+            _cli.console.print(json_output, soft_wrap=True, markup=False, emoji=False)
         else:
             _cli.console.print(Pretty(artifacts_json))
 

prefect/cli/block.py

@@ -344,9 +344,9 @@ async def block_delete(
             except ObjectNotFound:
                 exit_with_error(f"Block '{block_id}' not found!")
         elif slug is not None:
-            if "/" not in slug:
+            if slug.count("/") != 1:
                 exit_with_error(
-                    f"{slug!r} is not valid. Slug must contain a '/', e.g. 'json/my-json-block'"
+                    f"{slug!r} is not valid. Slug must contain exactly one '/', e.g. 'json/my-json-block'"
                 )
             block_type_slug, block_document_name = slug.split("/")
             try:
@@ -447,9 +447,9 @@ async def block_inspect(
             except ObjectNotFound:
                 exit_with_error(f"Block '{block_id}' not found!")
         elif slug is not None:
-            if "/" not in slug:
+            if slug.count("/") != 1:
                 exit_with_error(
-                    f"{slug!r} is not valid. Slug must contain a '/', e.g. 'json/my-json-block'"
+                    f"{slug!r} is not valid. Slug must contain exactly one '/', e.g. 'json/my-json-block'"
                 )
             block_type_slug, block_document_name = slug.split("/")
             try:

prefect/concurrency/AGENTS.md

@@ -18,7 +18,7 @@ Does NOT define concurrency limits (server-side in `server/`). Does NOT handle t
 
 **`raise_on_lease_renewal_failure` (public parameter):** Controls lease renewal failure behavior independently from `strict`. When `None` (default), falls back to the value of `strict` for backward compatibility. Set to `False` to let long-running tasks continue even if a transient renewal error occurs; set to `True` to terminate immediately on renewal failure regardless of `strict`. This means `strict=True, raise_on_lease_renewal_failure=False` gives strict slot acquisition but non-fatal renewal failures, and vice versa.
 
-**Lease renewal:** `_leases.py` starts a daemon thread that renews immediately on entry, then waits for 75% of `lease_duration` between renewals. The renewal thread uses the standard `get_client(sync_client=True)` path and retries each renewal up to 3 times with exponential backoff. If all attempts fail, a shared failure handler either cancels execution with the existing sync/async cancel scope or logs a warning, depending on `raise_on_lease_renewal_failure`.
+**Lease renewal:** `_leases.py` starts a daemon thread that renews immediately on entry, then waits for 75% of `lease_duration` between renewals. The renewal thread uses the standard `get_client(sync_client=True)` path and retries each renewal up to 3 times with exponential backoff. If all attempts fail, a shared failure handler either cancels execution with the existing sync/async cancel scope or logs a warning, depending on `raise_on_lease_renewal_failure`. Exception: when a `flow_run_id` is given and renewal fails with `410 Gone`, the thread reads the flow run and stops without cancelling if its server state is final, because the terminal transition releases the lease before the engine receives the proposal response.
 
 **Sync/async lockstep invariant:** `asyncio.py`/`sync.py` and `_asyncio.py`/`_sync.py` are parallel implementations. Any behavior change to one must be mirrored in the other.
 

prefect/concurrency/_leases.py

@@ -4,6 +4,9 @@ from contextlib import asynccontextmanager, contextmanager
 from typing import TYPE_CHECKING, AsyncGenerator, Generator
 from uuid import UUID
 
+import httpx
+
+from prefect._internal.compatibility.starlette import status
 from prefect._internal.concurrency.cancellation import (
     AsyncCancelScope,
     WatcherThreadCancelScope,
@@ -90,6 +93,45 @@ def _lease_renewal_loop(
             stop_event.wait(lease_duration * _RENEWAL_FRACTION)
 
 
+def _lease_released_by_final_flow_run(
+    exc: BaseException, lease_id: UUID, flow_run_id: UUID | None
+) -> bool:
+    """
+    Check whether a lease renewal failure is explained by the flow run having
+    reached a final state on the server, which releases the lease before the
+    engine has received the response to its terminal state proposal.
+    """
+    if flow_run_id is None or not (
+        isinstance(exc, httpx.HTTPStatusError)
+        and exc.response.status_code == status.HTTP_410_GONE
+    ):
+        return False
+
+    logger = get_logger("concurrency")
+    try:
+        with get_client(sync_client=True) as client:
+            flow_run = client.read_flow_run(flow_run_id)
+    except Exception:
+        logger.debug(
+            "Failed to read flow run %s after its concurrency lease was not found",
+            flow_run_id,
+            exc_info=True,
+        )
+        return False
+
+    if flow_run.state is None or not flow_run.state.is_final():
+        return False
+
+    logger.debug(
+        "Concurrency lease %s was released because flow run %s reached final "
+        "state %s; stopping lease renewal",
+        lease_id,
+        flow_run_id,
+        flow_run.state.type.value,
+    )
+    return True
+
+
 def _handle_lease_renewal_failure(
     exc: BaseException,
     lease_id: UUID,
@@ -139,6 +181,7 @@ def _start_lease_renewal_thread(
     raise_on_lease_renewal_failure: bool,
     suppress_warnings: bool,
     cancel_scope: AsyncCancelScope | WatcherThreadCancelScope,
+    flow_run_id: UUID | None = None,
 ) -> tuple[threading.Event, threading.Thread]:
     stop_event = threading.Event()
     renewal_ctx = contextvars.copy_context()
@@ -152,14 +195,17 @@ def _start_lease_renewal_thread(
         try:
             _lease_renewal_loop(lease_id, lease_duration, stop_event)
         except Exception as exc:
-            if not stop_event.is_set():
-                _handle_lease_renewal_failure(
-                    exc,
-                    lease_id,
-                    raise_on_lease_renewal_failure,
-                    suppress_warnings,
-                    cancel_scope,
-                )
+            if stop_event.is_set() or _lease_released_by_final_flow_run(
+                exc, lease_id, flow_run_id
+            ):
+                return
+            _handle_lease_renewal_failure(
+                exc,
+                lease_id,
+                raise_on_lease_renewal_failure,
+                suppress_warnings,
+                cancel_scope,
+            )
 
     def renewal_loop() -> None:
         renewal_ctx.run(_run_with_failure_handler)
@@ -179,6 +225,7 @@ def maintain_concurrency_lease(
     lease_duration: float,
     raise_on_lease_renewal_failure: bool = False,
     suppress_warnings: bool = False,
+    flow_run_id: UUID | None = None,
 ) -> Generator[None, None, None]:
     """
     Maintain a concurrency lease for the given lease ID.
@@ -187,6 +234,9 @@ def maintain_concurrency_lease(
         lease_id: The ID of the lease to maintain.
         lease_duration: The duration of the lease in seconds.
         raise_on_lease_renewal_failure: A boolean specifying whether to raise an error if the lease renewal fails.
+        flow_run_id: The ID of the flow run holding the lease. When the lease is
+            not found and this flow run is final on the server, the lease was
+            released by the terminal transition and execution is not terminated.
     """
     with WatcherThreadCancelScope() as cancel_scope:
         stop_event: threading.Event | None = None
@@ -198,6 +248,7 @@ def maintain_concurrency_lease(
                 raise_on_lease_renewal_failure,
                 suppress_warnings,
                 cancel_scope,
+                flow_run_id,
             )
             yield
         finally:
@@ -213,6 +264,7 @@ async def amaintain_concurrency_lease(
     lease_duration: float,
     raise_on_lease_renewal_failure: bool = False,
     suppress_warnings: bool = False,
+    flow_run_id: UUID | None = None,
 ) -> AsyncGenerator[None, None]:
     """
     Maintain a concurrency lease for the given lease ID.
@@ -221,6 +273,9 @@ async def amaintain_concurrency_lease(
         lease_id: The ID of the lease to maintain.
         lease_duration: The duration of the lease in seconds.
         raise_on_lease_renewal_failure: A boolean specifying whether to raise an error if the lease renewal fails.
+        flow_run_id: The ID of the flow run holding the lease. When the lease is
+            not found and this flow run is final on the server, the lease was
+            released by the terminal transition and execution is not terminated.
     """
     with AsyncCancelScope() as cancel_scope:
         stop_event: threading.Event | None = None
@@ -231,6 +286,7 @@ async def amaintain_concurrency_lease(
                 raise_on_lease_renewal_failure,
                 suppress_warnings,
                 cancel_scope,
+                flow_run_id,
             )
             yield
         finally:

prefect/flow_engine.py

@@ -1137,7 +1137,10 @@ class FlowRunEngine(BaseFlowRunEngine[P, R]):
             if lease_id := self.state.state_details.deployment_concurrency_lease_id:
                 stack.enter_context(
                     maintain_concurrency_lease(
-                        lease_id, 300, raise_on_lease_renewal_failure=True
+                        lease_id,
+                        300,
+                        raise_on_lease_renewal_failure=True,
+                        flow_run_id=self.flow_run.id,
                     )
                 )
 
@@ -1865,7 +1868,10 @@ class AsyncFlowRunEngine(BaseFlowRunEngine[P, R]):
             if lease_id := self.state.state_details.deployment_concurrency_lease_id:
                 await stack.enter_async_context(
                     amaintain_concurrency_lease(
-                        lease_id, 300, raise_on_lease_renewal_failure=True
+                        lease_id,
+                        300,
+                        raise_on_lease_renewal_failure=True,
+                        flow_run_id=self.flow_run.id,
                     )
                 )
 

prefect/flow_runs.py

@@ -133,6 +133,7 @@ async def wait_for_flow_run(
     )
 
     with anyio.move_on_after(timeout):
+        received_final_event = False
         async with get_events_subscriber(filter=event_filter) as subscriber:
             flow_run = await client.read_flow_run(flow_run_id)
             if flow_run.state and flow_run.state.is_final():
@@ -151,7 +152,19 @@ async def wait_for_flow_run(
                     logger.info(f"Flow run is in state {state.name!r}")
 
                 if state.is_final():
-                    return await client.read_flow_run(flow_run_id)
+                    received_final_event = True
+                    break
+
+        if received_final_event:
+            # A final state event can be delivered before the state change is
+            # visible to API reads, so read until the run reports a final state
+            poll_delay = 0.1
+            while True:
+                flow_run = await client.read_flow_run(flow_run_id)
+                if flow_run.state and flow_run.state.is_final():
+                    return flow_run
+                await anyio.sleep(poll_delay)
+                poll_delay = min(poll_delay * 2, 5)
 
     raise FlowRunWaitTimeout(
         f"Flow run with ID {flow_run_id} exceeded watch timeout of {timeout} seconds"

prefect/flows.py

@@ -706,11 +706,11 @@ class Flow(Generic[P, R]):
                 from fastapi.encoders import jsonable_encoder
 
                 serialized_parameters[key] = jsonable_encoder(value)
-            except (TypeError, ValueError, RecursionError):
-                # `jsonable_encoder` recurses into unknown objects with no cycle
-                # or depth limit, so a deeply-nested or self-referential value
-                # raises `RecursionError`. Treat it like any other unserializable
-                # value and fall back to the placeholder below.
+            except Exception:
+                # `jsonable_encoder` can raise arbitrary errors for values it
+                # cannot encode (e.g. `RecursionError` for self-referential
+                # objects, or any error from `__reduce__` while deep-copying a
+                # dataclass). Fall back to the placeholder below.
                 logger.debug(
                     f"Parameter {key!r} for flow {self.name!r} is unserializable. "
                     f"Type {type(value).__name__!r} and will not be stored "

prefect/server/AGENTS.md

@@ -54,7 +54,7 @@ alembic_revision("description")      # Create a new migration
 - **Deployment concurrency has a TLA+ model.** Before changing its behavior,
   review `formal/tla/deployment-concurrency/README.md` and its mapped regressions.
 
-- **Pydantic v2 treats null JSON fields as explicitly set.** When a worker sends a state update with `field: null`, Pydantic v2 sets that field to `None`, silently overwriting any existing value. To preserve `state_details` fields across transitions (e.g. `deployment_concurrency_lease_id`), add a `FlowRunUniversalTransform` to `CoreFlowPolicy` that copies the field forward when the proposed state has `None`. See `PreserveDeploymentConcurrencyLeaseId` in `orchestration/core_policy.py` as the canonical pattern. Any new field added to `state_details` that workers may omit faces this same risk.
+- **Pydantic v2 treats null JSON fields as explicitly set.** When a worker sends a state update with `field: null`, Pydantic v2 sets that field to `None`, silently overwriting any existing value. To preserve `state_details` fields across transitions (e.g. `deployment_concurrency_lease_id`), add a `FlowRunUniversalTransform` to both `CoreFlowPolicy` and `MinimalFlowPolicy` (used for `force=True`, otherwise forced transitions drop the field) that copies the field forward when the proposed state has `None`. See `PreserveDeploymentConcurrencyLeaseId` in `orchestration/core_policy.py` as the canonical pattern. Any new field added to `state_details` that workers may omit faces this same risk.
 
 - **`update_deployment` uses `model_fields_set` to distinguish explicit `None` from "not provided" for `work_pool_name`.** In `models/deployments.py`, when `deployment.work_pool_name is None` AND `"work_pool_name" in deployment.model_fields_set`, the work queue association is cleared (`work_queue_id = None`). If `work_pool_name` is simply absent from `model_fields_set`, the existing work pool association is left intact. This is the intentional counterpart to the Pydantic v2 null-overwrite pitfall above — here the explicit `model_fields_set` entry signals *desired* clearing. `RunnerDeployment` factory methods omit `None`-valued work pool fields from constructor kwargs; `Runner.add_flow()` then post-assigns `None` to opt into clearing. Follow this same pattern for any future field that should distinguish "clear it" from "leave it alone."
 
@@ -68,11 +68,12 @@ alembic_revision("description")      # Create a new migration
 
 - **CANCELLING state transitions must call `maybe_schedule_cancelling_timeout_check_for_state()`.** Any new API path that accepts a CANCELLING transition should call this after `SetStateStatus.ACCEPT`; failures are intentionally non-fatal — log and swallow, do not raise. `cancel_subflow_run` deliberately skips subflows already in CANCELLING to preserve their existing timeout deadline — this guard has no inline comment but removing it resets the deadline. The `ensure_cancelling_timeout_checks` perpetual service re-seeds missed checks on server restart.
 
-- **`record_bulk_task_run_events` batches must be sorted by conflict key before upserting.** In `services/task_run_recorder.py`, task runs are sorted by conflict key — natural key `(flow_run_id, task_key, dynamic_key)` when available, otherwise `("id", task_run_id)` — before batching into upsert groups. This enforces deterministic row-level lock acquisition order across concurrent recorder instances; removing or reordering the sort causes deadlocks. Events that collide on either `id` or natural key are coalesced via union-find to a single canonical ID — the input `TaskRun.id` and `state.state_details.task_run_id` are mutated in-place to that canonical value. Any refactor that re-batches or re-merges must re-sort by conflict key. The function retries once internally on `IntegrityError` to handle TOCTOU races between concurrent recorders, then re-raises on a second failure — callers must catch `IntegrityError` or re-queue the batch.
+- **`record_bulk_task_run_events` batches must be sorted by conflict key before upserting.** In `services/task_run_recorder.py`, task runs are sorted by conflict key — natural key `(flow_run_id, task_key, dynamic_key)` when available, otherwise `("id", task_run_id)` — before batching into upsert groups. This enforces deterministic row-level lock acquisition order across concurrent recorder instances; removing or reordering the sort causes deadlocks. Events that collide on either `id` or natural key are coalesced via union-find to a single canonical ID — the input `TaskRun.id` and `state.state_details.task_run_id` are mutated in-place to that canonical value. Any refactor that re-batches or re-merges must re-sort by conflict key. The function retries once internally on `IntegrityError` to handle TOCTOU races between concurrent recorders, then re-raises on a second failure — callers must catch `IntegrityError` or re-queue the batch. Events whose `flow_run_id` no longer exists are dropped with a warning before the upsert, since one foreign-key violation would otherwise fail the whole batch (#23663).
 
 - **`_segment_task_runs_for_upsert` must keep emitting contiguous blocks of its input.** That property, not the batch key, is what preserves the lock ordering above, so a new condition may only *close* a segment earlier — never reorder rows. Each existing condition guards a failure that is silent on at least one dialect; see its docstring and `_fillable_columns` before relaxing one. Absent keys are `NULL`-filled and restored with `coalesce(excluded.col, task_run.col)`, applied to exactly the columns that were filled — a payload asking to write a real `NULL` keeps a plain `excluded.col`, and the batch splits rather than let one statement mean both. This is also why `flow_run_id` must stay out of `_fillable_columns`: coalescing it would break detach.
 
 - **`docket.add()` in monitor loops must pass a per-entity `key=` to prevent duplicate enqueues.** Without it, repeated iterations over the same pending entity enqueue duplicate tasks — causing duplicate state transitions. Format: `"<service-name>:<entity-id>"`, e.g., `"mark-flow-run-late:{run.id}"` in `services/late_runs.py`.
+  Keyed `add()` can suppress later enqueues while earlier work remains known to Docket, including stranded work awaiting redelivery after a Redis disruption. Use `replace()` only when superseding the pending schedule and arguments is safe (see `schedule_readiness_update` in `api/workers.py`). Choose keys by logical work: independent updates that must both run need distinct keys, while updates intended to supersede one another share a key. Replacement does not synchronously cancel already-running work; `redelivery_timeout` controls reclamation eligibility, not key expiration.
 
 - **`TaskRunRecorder` and `EventPersister` share a buffered-consumer shape; keep their failure handling in sync.** Both (`services/task_run_recorder.py`, `events/services/event_persister.py`) queue messages and flush from a never-awaited `flush_periodically` task, so any exception escaping that loop silently stops flushing for the life of the service (#22441, #23308). Database drivers can raise `CancelledError` during outages, so the loop exits on cancellation only after the shutdown path sets its `stopping` flag. Mirror retry/drop/cancellation fixes in both.
 

prefect/server/api/deployments.py

@@ -20,7 +20,7 @@ from prefect.server.api.validation import (
     validate_job_variables_for_deployment,
     validate_job_variables_for_deployment_flow_run,
 )
-from prefect.server.api.workers import WorkerLookups
+from prefect.server.api.workers import WorkerLookups, schedule_readiness_update
 from prefect.server.database import PrefectDBInterface, provide_database_interface
 from prefect.server.exceptions import MissingVariableError, ObjectNotFoundError
 from prefect.server.models.deployments import mark_deployments_ready
@@ -656,10 +656,16 @@ async def get_scheduled_flow_runs_for_deployments(
             for orm_flow_run in orm_flow_runs
         ]
 
+        becoming_ready = await models.deployments.has_not_ready_deployments(
+            session=session, deployment_ids=deployment_ids
+        )
+
     sorted_deployment_ids = ",".join(str(d) for d in sorted(deployment_ids))
-    await docket.add(
+    await schedule_readiness_update(
+        docket,
         mark_deployments_ready,
         key=f"mark_deployments_ready:deployments:{sorted_deployment_ids}",
+        becoming_ready=becoming_ready,
     )(
         deployment_ids=deployment_ids,
     )

prefect/server/api/flow_runs.py

@@ -11,6 +11,7 @@ from urllib.parse import quote
 from uuid import UUID
 
 import orjson
+import pydantic_core
 import sqlalchemy as sa
 from docket import Depends as DocketDepends
 from docket import Docket, Retry
@@ -69,6 +70,21 @@ logger: "logging.Logger" = get_logger("server.api")
 router: PrefectRouter = PrefectRouter(prefix="/flow_runs", tags=["Flow Runs"])
 
 
+def _json_response(content: Any) -> Response:
+    """
+    Serialize already JSON-compatible `content` into a JSON response.
+
+    orjson is much faster than the FastAPI encoder, but it rejects integers
+    outside of the 64-bit range. Flow run parameters are arbitrary user data, so
+    fall back to pydantic-core, which supports arbitrarily large integers.
+    """
+    try:
+        body = orjson.dumps(content)
+    except TypeError:
+        body = pydantic_core.to_json(content)
+    return Response(content=body, media_type="application/json")
+
+
 def _get_request_docket(request: Request) -> Docket | None:
     return getattr(request.app.state, "docket", None)
 
@@ -621,10 +637,7 @@ async def read_flow_runs(
             ).model_dump(mode="json")
             for fr in db_flow_runs
         ]
-        return Response(
-            content=orjson.dumps(encoded),
-            media_type="application/json",
-        )
+        return _json_response(encoded)
 
 
 @router.delete("/{id:uuid}", status_code=status.HTTP_204_NO_CONTENT)
@@ -1047,10 +1060,7 @@ async def paginate_flow_runs(
         page=page,
     ).model_dump(mode="json")
 
-    return Response(
-        content=orjson.dumps(response),
-        media_type="application/json",
-    )
+    return _json_response(response)
 
 
 FLOW_RUN_LOGS_DOWNLOAD_PAGE_LIMIT = 1000

prefect/server/api/server.py

@@ -11,7 +11,9 @@ import contextlib
 import gc
 import hmac
 import importlib.metadata
+import json
 import logging
+import math
 import mimetypes
 import os
 import random
@@ -292,16 +294,34 @@ async def validation_exception_handler(
     """Provide a detailed message for request validation errors."""
     return JSONResponse(
         status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
-        content=jsonable_encoder(
-            {
-                "exception_message": "Invalid request received.",
-                "exception_detail": exc.errors(),
-                "request_body": exc.body,
-            }
+        content=_json_safe(
+            jsonable_encoder(
+                {
+                    "exception_message": "Invalid request received.",
+                    "exception_detail": exc.errors(),
+                    "request_body": exc.body,
+                }
+            )
         ),
     )
 
 
+def _json_safe(value: Any) -> Any:
+    """
+    Replace values that cannot be rendered as strict JSON, such as `NaN` or strings
+    containing unpaired surrogates, so that invalid input can be echoed back.
+    """
+    if isinstance(value, float) and not math.isfinite(value):
+        return json.dumps(value)
+    if isinstance(value, str):
+        return value.encode("utf-8", "backslashreplace").decode("utf-8")
+    if isinstance(value, dict):
+        return {_json_safe(k): _json_safe(v) for k, v in value.items()}
+    if isinstance(value, list):
+        return [_json_safe(v) for v in value]
+    return value
+
+
 async def integrity_exception_handler(request: Request, exc: Exception) -> JSONResponse:
     """Capture database integrity errors."""
     logger.error("Encountered exception in request:", exc_info=True)
@@ -317,7 +337,20 @@ async def integrity_exception_handler(request: Request, exc: Exception) -> JSONR
     )
 
 
+def is_invalid_data_exception(exc: Exception) -> bool:
+    """
+    Whether the database rejected a value as invalid (SQLSTATE class 22, "data
+    exception"), e.g. a NUL character or `NaN` in a PostgreSQL JSON value.
+    """
+    orig = exc.orig if isinstance(exc, sqlalchemy.exc.DBAPIError) else exc
+    sqlstate = getattr(orig, "sqlstate", None)
+    return isinstance(sqlstate, str) and sqlstate.startswith("22")
+
+
 def is_client_retryable_exception(exc: Exception) -> bool:
+    if is_invalid_data_exception(exc):
+        return False
+
     if isinstance(exc, sqlalchemy.exc.OperationalError) and isinstance(
         exc.orig, sqlite3.OperationalError
     ):
@@ -399,8 +432,29 @@ async def custom_internal_exception_handler(
     """
     Log a detailed exception for internal server errors before returning.
 
-    Send 503 for errors clients can retry on.
+    Send 422 for values the database rejected and 503 for errors clients can retry on.
     """
+    if is_invalid_data_exception(exc):
+        logger.warning("Database rejected a value in request:", exc_info=True)
+        return JSONResponse(
+            content={
+                "exception_message": "Invalid request received.",
+                "exception_detail": [
+                    {
+                        "type": "value_error",
+                        "loc": ["body"],
+                        "msg": (
+                            "The database rejected a value in the request. This "
+                            "usually means a string contains a NUL character or a "
+                            "JSON value contains NaN or Infinity. See server logs "
+                            "for details."
+                        ),
+                    }
+                ],
+            },
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+        )
+
     if is_client_retryable_exception(exc):
         if PREFECT_API_LOG_RETRYABLE_ERRORS.value():
             logger.error("Encountered retryable exception in request:", exc_info=True)

prefect/server/api/work_queues.py

@@ -18,6 +18,7 @@ from sqlalchemy.exc import IntegrityError
 import prefect.server.api.dependencies as dependencies
 import prefect.server.models as models
 import prefect.server.schemas as schemas
+from prefect.server.api.workers import schedule_readiness_update
 from prefect.server.database import (
     PrefectDBInterface,
     provide_database_interface,
@@ -189,19 +190,32 @@ async def read_work_queue_runs(
     if x_prefect_ui:
         return flow_runs
 
-    await docket.add(
+    work_queue_becoming_ready = work_queue.status == WorkQueueStatus.NOT_READY
+    if work_queue_becoming_ready:
+        deployments_becoming_ready = True
+    else:
+        async with db.session_context() as session:
+            deployments_becoming_ready = (
+                await models.deployments.has_not_ready_deployments(
+                    session=session, work_queue_ids=[work_queue_id]
+                )
+            )
+
+    await schedule_readiness_update(
+        docket,
         mark_work_queues_ready,
         key=f"mark_work_queues_ready:{work_queue_id}",
+        becoming_ready=work_queue_becoming_ready,
     )(
         polled_work_queue_ids=[work_queue_id],
-        ready_work_queue_ids=(
-            [work_queue_id] if work_queue.status == WorkQueueStatus.NOT_READY else []
-        ),
+        ready_work_queue_ids=[work_queue_id] if work_queue_becoming_ready else [],
     )
 
-    await docket.add(
+    await schedule_readiness_update(
+        docket,
         mark_deployments_ready,
         key=f"mark_deployments_ready:work_queue:{work_queue_id}",
+        becoming_ready=deployments_becoming_ready,
     )(
         work_queue_ids=[work_queue_id],
     )