prefect-client 3.6.30.dev3__tar.gz → 3.6.30.dev4__tar.gz

{prefect_client-3.6.30.dev3 → prefect_client-3.6.30.dev4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: prefect-client
-Version: 3.6.30.dev3
+Version: 3.6.30.dev4
 Summary: Workflow orchestration and management.
 Project-URL: Changelog, https://github.com/PrefectHQ/prefect/releases
 Project-URL: Documentation, https://docs.prefect.io

{prefect_client-3.6.30.dev3 → prefect_client-3.6.30.dev4}/src/prefect/AGENTS.md

@@ -10,6 +10,7 @@ There is no formal public/private boundary beyond the `_` prefix convention. Mod
 
 - `flows.py` / `tasks.py` — `@flow` and `@task` decorator definitions. The primary user-facing API.
 - `flow_engine.py` / `task_engine.py` — Async execution engines that orchestrate runs. **Critical invariant:** both sync and async engine paths must be kept in lockstep. Changes to one must be mirrored in the other.
+- `flow_engine.py` is also a subprocess entrypoint: `python -m prefect.flow_engine <entrypoint>`. Reads `PREFECT__FLOW_RUN_ID` from the environment; accepts exactly one argument (the flow entrypoint path or module ref). Workers and infrastructure use this to launch flow runs as isolated subprocesses. Plain functions not decorated with `@flow` are auto-converted to flows at this entry point.
 - `states.py` — State objects and transition logic
 - `results.py` — Result persistence and retrieval
 - `futures.py` — `PrefectFuture` for async task results
@@ -20,12 +21,15 @@ There is no formal public/private boundary beyond the `_` prefix convention. Mod
 ## Key Contracts
 
 - **Engine ordering matters.** The engines apply features (retries, caching, result persistence, transactions) in a specific order. Changing the order or forgetting a feature in one engine path is the most common source of breakage.
+- **Set run metadata before state transitions.** Any run attribute that event subscribers observe at the moment of a state transition (e.g., custom `flow_run_name`) must be persisted to the server *before* `set_state()` is called for that transition. Metadata set after the call appears stale to subscribers of that state event.
 - **Sync and async must stay in sync.** Both `flow_engine.py` and `task_engine.py` have sync and async paths. Any behavior change must be applied to both.
 - **Use `whenever` compat helpers for all datetime conversions on Python 3.13+.** `types/_datetime.py` provides `now()`, `_whenever_to_stdlib()`, `_whenever_zdt_from_py()`, and `_whenever_pdt_from_py()` to abstract over API differences between whenever 0.7.x–0.9.x and ≥ 0.10.0. The `_WHENEVER_NEW_API` flag (True when whenever ≥ 0.10.0) guards version-specific code. Never call `DateTime.now()` or `pendulum.now()` directly — use `now()` from `types/_datetime.py` instead. Never call `ZonedDateTime.from_py_datetime()`, `PlainDateTime.from_py_datetime()`, or `.py_datetime()` directly — use the helpers instead. Violations will silently break on whichever whenever version the helpers weren't written for.
 - **Flow and task engines advance state differently.** The flow engine makes blocking API calls to the server to propose and advance states. The task engine emits `prefect.task_run.*` events (delivered via WebSockets) but advances state locally through `set_state` calls with polling/backoff — do not assume the two engines work the same way.
 - **Flow state transitions go through the server.** The flow engine proposes states to the server, which accepts or rejects them via orchestration rules. The task engine, by contrast, manages state transitions locally via `set_state` and emits `prefect.task_run.*` events — it does not propose states to the server.
 - **`ProcessPoolTaskRunner` requires picklable data across subprocess boundaries.** `PrefectFuture` objects are not picklable and cannot be passed to worker subprocesses. Any `wait_for` futures must be waited on in the parent process and converted to `State` objects before submission. The subprocess task engine handles `State` objects via `resolve_to_final_result`, which raises `UpstreamTaskError` for non-completed upstreams.
 - **`ProcessPoolTaskRunner` instances may be deserialized without `__init__` running.** When a runner is pickled and restored in a subprocess, `__init__` is not called, so instance attributes set there may be absent. Access any such attribute via `getattr(self, "_attr", default)` rather than `self._attr` directly — both in the property getter and in `duplicate()`. The `subprocess_message_processor_factories` property demonstrates the required pattern.
+- **`ThreadPoolTaskRunner` is cloudpickled when a flow run is dispatched to a subprocess.** `threading.Lock` and other non-picklable thread primitives must be dropped in `__getstate__` and rebuilt in `__setstate__`. Any new instance state added to this class must be evaluated for picklability.
+- **Nested task submissions on a bounded `ThreadPoolTaskRunner` can deadlock.** When a worker task submits children and blocks on `.result()` while all `max_workers` threads are busy, the pool starves. `_warn_if_nested_submit_would_deadlock` detects this and emits a one-time warning — preserve this detection when changing pool management.
 - **`ResultRecordMetadata` tolerates unknown serializer types.** When loading persisted metadata, an unrecognized serializer `type` is converted to an `UnknownSerializer` placeholder rather than raising `ValidationError`. This allows inspecting result metadata when the serializer implementation is unavailable in the current environment. However, known serializer types with invalid fields still raise `ValidationError`. `UnknownSerializer.dumps()` and `UnknownSerializer.loads()` raise `RuntimeError` — the tolerance is for inspection only, not actual serialization/deserialization.
 
 ## Logging

prefect_client-3.6.30.dev4/src/prefect/_build_info.py

@@ -0,0 +1,5 @@
+# Generated by versioningit
+__version__ = "3.6.30.dev4"
+__build_date__ = "2026-05-06 08:56:13.081322+00:00"
+__git_commit__ = "2b9d462118706889dfeee26df45a1b6108491070"
+__dirty__ = False

{prefect_client-3.6.30.dev3 → prefect_client-3.6.30.dev4}/src/prefect/flow_engine.py

@@ -8,6 +8,7 @@ import multiprocessing
 import multiprocessing.context
 import os
 import signal
+import sys
 import threading
 import time
 from contextlib import (
@@ -66,7 +67,7 @@ from prefect.context import (
     hydrated_context,
     serialize_context,
 )
-from prefect.engine import handle_engine_signals
+from prefect.engine import _drive_run_flow_result, handle_engine_signals
 from prefect.events.related import RelatedResource, tags_as_related_resources
 from prefect.events.utilities import emit_event
 from prefect.exceptions import (
@@ -92,6 +93,8 @@ from prefect.logging.loggers import (
 )
 from prefect.results import (
     ResultStore,
+    _aget_default_persist_result,
+    _get_default_persist_result,
     get_result_store,
     should_persist_result,
 )
@@ -109,6 +112,7 @@ from prefect.states import (
     exception_to_failed_state,
     return_value_to_state,
 )
+from prefect.telemetry._metrics import RunMetrics
 from prefect.telemetry.run_telemetry import (
     LABELS_TRACEPARENT_KEY,
     TRACEPARENT_KEY,
@@ -142,6 +146,7 @@ P = ParamSpec("P")
 R = TypeVar("R")
 
 MINIMUM_HEARTBEAT_INTERVAL = 30
+_engine_logger = get_logger("engine")
 _CONTROL_CHANNEL_ENV_KEYS = frozenset(
     {"PREFECT__CONTROL_PORT", "PREFECT__CONTROL_TOKEN"}
 )
@@ -220,15 +225,18 @@ def load_flow_run(flow_run_id: UUID) -> FlowRun:
     return flow_run
 
 
+def _load_flow_from_runtime_entrypoint(entrypoint: str) -> Flow[..., Any]:
+    try:
+        return load_flow_from_entrypoint(entrypoint, use_placeholder_flow=False)
+    except MissingFlowError:
+        return load_function_and_convert_to_flow(entrypoint)
+
+
 def load_flow(flow_run: FlowRun) -> Flow[..., Any]:
     entrypoint = os.environ.get("PREFECT__FLOW_ENTRYPOINT")
 
     if entrypoint:
-        # we should not accept a placeholder flow at runtime
-        try:
-            flow = load_flow_from_entrypoint(entrypoint, use_placeholder_flow=False)
-        except MissingFlowError:
-            flow = load_function_and_convert_to_flow(entrypoint)
+        flow = _load_flow_from_runtime_entrypoint(entrypoint)
     else:
         flow = run_coro_as_sync(
             load_flow_from_flow_run(flow_run, use_placeholder_flow=False)
@@ -242,6 +250,51 @@ def load_flow_and_flow_run(flow_run_id: UUID) -> tuple[FlowRun, Flow[..., Any]]:
     return flow_run, flow
 
 
+def _run_flow_from_runtime_entrypoint(flow_run_id: UUID, entrypoint: str) -> None:
+    configure_from_env()
+
+    with handle_engine_signals(flow_run_id):
+        flow_run = load_flow_run(flow_run_id=flow_run_id)
+        run_logger = flow_run_logger(flow_run=flow_run)
+
+        try:
+            flow = _load_flow_from_runtime_entrypoint(entrypoint)
+        except Exception:
+            run_logger.error(
+                "Unexpected exception encountered when trying to load flow",
+                exc_info=True,
+            )
+            raise
+
+        with RunMetrics(flow_run, flow):
+            run_result = run_flow(flow, flow_run=flow_run, error_logger=run_logger)
+            _drive_run_flow_result(flow, run_result)
+
+
+def _main(argv: list[str] | None = None) -> int:
+    args = sys.argv[1:] if argv is None else argv
+    if len(args) != 1:
+        _engine_logger.error(
+            "Invalid flow entrypoint. Expected one argument; received: %s", args
+        )
+        return 1
+
+    flow_run_id_value = os.environ.get("PREFECT__FLOW_RUN_ID")
+    try:
+        flow_run_id = UUID(flow_run_id_value) if flow_run_id_value else None
+    except ValueError:
+        flow_run_id = None
+
+    if flow_run_id is None:
+        _engine_logger.error(
+            "Invalid flow run id. Expected PREFECT__FLOW_RUN_ID to contain a UUID."
+        )
+        return 1
+
+    _run_flow_from_runtime_entrypoint(flow_run_id, args[0])
+    return 0
+
+
 @contextmanager
 def _send_heartbeats(
     engine: "BaseFlowRunEngine[Any, Any]",
@@ -602,6 +655,32 @@ class FlowRunEngine(BaseFlowRunEngine[P, R]):
                 )
                 self.short_circuit = True
 
+        if (
+            not self.short_circuit
+            and not self._flow_run_name_set
+            and self.flow.flow_run_name
+        ):
+            if self.flow_run is None:
+                raise ValueError("Flow run not set")
+
+            with FlowRunContext(
+                flow=self.flow,
+                flow_run=self.flow_run,
+                parameters=self.parameters,
+                client=self.client,
+                task_runner=self.flow.task_runner,
+                result_store=get_result_store().update_for_flow(self.flow, _sync=True),
+            ):
+                flow_run_name = resolve_custom_flow_run_name(
+                    flow=self.flow, parameters=self.parameters or {}
+                )
+            self.client.set_flow_run_name(
+                flow_run_id=self.flow_run.id, name=flow_run_name
+            )
+            self.flow_run.name = flow_run_name
+            self._flow_run_name_set = True
+            self._telemetry.update_run_name(name=flow_run_name)
+
         new_state = Running()
         state = self.set_state(new_state)
         while state.is_pending():
@@ -927,6 +1006,12 @@ class FlowRunEngine(BaseFlowRunEngine[P, R]):
             if log_prints:
                 stack.enter_context(patch_print())
             task_runner = stack.enter_context(self.flow.task_runner.duplicate())
+            result_store = get_result_store().update_for_flow(self.flow, _sync=True)
+            persist_result = (
+                self.flow.persist_result
+                if self.flow.persist_result is not None
+                else _get_default_persist_result()
+            )
             stack.enter_context(
                 FlowRunContext(
                     flow=self.flow,
@@ -934,13 +1019,9 @@ class FlowRunEngine(BaseFlowRunEngine[P, R]):
                     flow_run=self.flow_run,
                     parameters=self.parameters,
                     client=client,
-                    result_store=get_result_store().update_for_flow(
-                        self.flow, _sync=True
-                    ),
+                    result_store=result_store,
                     task_runner=task_runner,
-                    persist_result=self.flow.persist_result
-                    if self.flow.persist_result is not None
-                    else should_persist_result(),
+                    persist_result=persist_result,
                 )
             )
             # Set deployment context vars only if this is the top-level deployment run
@@ -1248,6 +1329,32 @@ class AsyncFlowRunEngine(BaseFlowRunEngine[P, R]):
                 )
                 self.short_circuit = True
 
+        if (
+            not self.short_circuit
+            and not self._flow_run_name_set
+            and self.flow.flow_run_name
+        ):
+            if self.flow_run is None:
+                raise ValueError("Flow run not set")
+
+            with FlowRunContext(
+                flow=self.flow,
+                flow_run=self.flow_run,
+                parameters=self.parameters,
+                client=self.client,
+                task_runner=self.flow.task_runner,
+                result_store=get_result_store().update_for_flow(self.flow, _sync=True),
+            ):
+                flow_run_name = resolve_custom_flow_run_name(
+                    flow=self.flow, parameters=self.parameters or {}
+                )
+            await self.client.set_flow_run_name(
+                flow_run_id=self.flow_run.id, name=flow_run_name
+            )
+            self.flow_run.name = flow_run_name
+            self._flow_run_name_set = True
+            self._telemetry.update_run_name(name=flow_run_name)
+
         new_state = Running()
         state = await self.set_state(new_state)
         while state.is_pending():
@@ -1566,6 +1673,12 @@ class AsyncFlowRunEngine(BaseFlowRunEngine[P, R]):
             if log_prints:
                 stack.enter_context(patch_print())
             task_runner = stack.enter_context(self.flow.task_runner.duplicate())
+            result_store = get_result_store().update_for_flow(self.flow, _sync=True)
+            persist_result = (
+                self.flow.persist_result
+                if self.flow.persist_result is not None
+                else await _aget_default_persist_result()
+            )
             stack.enter_context(
                 FlowRunContext(
                     flow=self.flow,
@@ -1573,13 +1686,9 @@ class AsyncFlowRunEngine(BaseFlowRunEngine[P, R]):
                     flow_run=self.flow_run,
                     parameters=self.parameters,
                     client=client,
-                    result_store=get_result_store().update_for_flow(
-                        self.flow, _sync=True
-                    ),
+                    result_store=result_store,
                     task_runner=task_runner,
-                    persist_result=self.flow.persist_result
-                    if self.flow.persist_result is not None
-                    else should_persist_result(),
+                    persist_result=persist_result,
                 )
             )
             # Set deployment context vars only if this is the top-level deployment run
@@ -2119,3 +2228,7 @@ def run_flow_in_subprocess(
     process.start()
 
     return process
+
+
+if __name__ == "__main__":
+    raise SystemExit(_main())

{prefect_client-3.6.30.dev3 → prefect_client-3.6.30.dev4}/src/prefect/flows.py

@@ -1886,10 +1886,14 @@ class Flow(Generic[P, R]):
             return_type=return_type,
         )
 
-    async def avisualize(self, *args: "P.args", **kwargs: "P.kwargs") -> None:
+    async def avisualize(
+        self,
+        *args: "P.args",
+        **kwargs: "P.kwargs",
+    ) -> None:
         """
-        Generates a graphviz object representing the current flow. In IPython notebooks,
-        it's rendered inline, otherwise in a new window as a PNG.
+        Generates a visualization representing the current flow. In IPython notebooks,
+        graphviz output is rendered inline, otherwise in a new window as a PNG.
 
         Raises:
             - ImportError: If `graphviz` isn't installed.
@@ -1920,7 +1924,6 @@ class Flow(Generic[P, R]):
                     self.fn(*args, **kwargs)
 
                 graph = build_task_dependencies(tracker)
-
                 visualize_task_dependencies(graph, self.name)
 
         except GraphvizImportError:
@@ -1945,10 +1948,14 @@ class Flow(Generic[P, R]):
             raise new_exception
 
     @async_dispatch(avisualize)
-    def visualize(self, *args: "P.args", **kwargs: "P.kwargs") -> None:
+    def visualize(
+        self,
+        *args: "P.args",
+        **kwargs: "P.kwargs",
+    ) -> None:
         """
-        Generates a graphviz object representing the current flow. In IPython notebooks,
-        it's rendered inline, otherwise in a new window as a PNG.
+        Generates a visualization representing the current flow. In IPython notebooks,
+        graphviz output is rendered inline, otherwise in a new window as a PNG.
 
         Raises:
             - ImportError: If `graphviz` isn't installed.
@@ -1980,7 +1987,6 @@ class Flow(Generic[P, R]):
                     self.fn(*args, **kwargs)
 
                 graph = build_task_dependencies(tracker)
-
                 visualize_task_dependencies(graph, self.name)
 
         except GraphvizImportError:
@@ -2004,6 +2010,112 @@ class Flow(Generic[P, R]):
             new_exception.__traceback__ = e.__traceback__
             raise new_exception
 
+    async def agenerate_mermaid_graph(
+        self,
+        *args: "P.args",
+        **kwargs: "P.kwargs",
+    ) -> str:
+        """
+        Generates a Mermaid flowchart diagram representing the structure of the current
+        flow and returns it as a string.
+
+        Returns:
+            A Mermaid `flowchart TD` diagram string.
+
+        Raises:
+            - FlowVisualizationError: If the flow can't be visualized for any other reason.
+        """
+        from prefect.utilities.visualization import (
+            FlowVisualizationError,
+            TaskVizTracker,
+            VisualizationUnsupportedError,
+            build_mermaid_dependencies,
+        )
+
+        if not PREFECT_TESTING_UNIT_TEST_MODE:
+            warnings.warn(
+                "`flow.generate_mermaid_graph()` will execute code inside of your flow"
+                " that is not decorated with `@task` or `@flow`."
+            )
+
+        try:
+            with TaskVizTracker() as tracker:
+                if self.isasync:
+                    await self.fn(*args, **kwargs)  # type: ignore[reportGeneralTypeIssues]
+                else:
+                    self.fn(*args, **kwargs)
+
+                return build_mermaid_dependencies(tracker)
+
+        except VisualizationUnsupportedError:
+            raise
+        except FlowVisualizationError:
+            raise
+        except Exception as e:
+            msg = (
+                "It's possible you are trying to visualize a flow that contains "
+                "code that directly interacts with the result of a task"
+                " inside of the flow. \nTry passing a `viz_return_value` "
+                "to the task decorator, e.g. `@task(viz_return_value=[1, 2, 3]).`"
+            )
+
+            new_exception = type(e)(str(e) + "\n" + msg)
+            new_exception.__traceback__ = e.__traceback__
+            raise new_exception
+
+    def generate_mermaid_graph(
+        self,
+        *args: "P.args",
+        **kwargs: "P.kwargs",
+    ) -> str:
+        """
+        Generates a Mermaid flowchart diagram representing the structure of the current
+        flow and returns it as a string.
+
+        Returns:
+            A Mermaid `flowchart TD` diagram string.
+
+        Raises:
+            - FlowVisualizationError: If the flow can't be visualized for any other reason.
+        """
+        from prefect.utilities.visualization import (
+            FlowVisualizationError,
+            TaskVizTracker,
+            VisualizationUnsupportedError,
+            build_mermaid_dependencies,
+        )
+
+        if not PREFECT_TESTING_UNIT_TEST_MODE:
+            warnings.warn(
+                "`flow.generate_mermaid_graph()` will execute code inside of your flow"
+                " that is not decorated with `@task` or `@flow`."
+            )
+
+        try:
+            with TaskVizTracker() as tracker:
+                if self.isasync:
+                    run_coro_as_sync(self.fn(*args, **kwargs))
+                else:
+                    self.fn(*args, **kwargs)
+
+                return build_mermaid_dependencies(tracker)
+
+        except VisualizationUnsupportedError:
+            raise
+        except FlowVisualizationError:
+            raise
+        except Exception as e:
+            msg = (
+                "It's possible you are trying to visualize a flow that contains "
+                "code that directly interacts with the result of a task"
+                " inside of the flow. \nTry passing a `viz_return_value` "
+                "to the task decorator, e.g. `@task(viz_return_value=[1, 2, 3]).`"
+            )
+
+            new_exception = type(e)(str(e) + "\n" + msg)
+            new_exception.__traceback__ = e.__traceback__
+            raise new_exception
+
 
 class FlowDecorator:
     @overload
@@ -2542,7 +2654,13 @@ class InfrastructureBoundFlow(Flow[P, R]):
             upload_bundle_to_storage,
         )
         from prefect.context import FlowRunContext, TagsContext
-        from prefect.results import get_result_store, resolve_result_storage
+        from prefect.results import (
+            _DefaultResultStorageSource,
+            _get_default_result_storage,
+            _result_storage_is_configured_for_remote_retrieval,
+            get_result_store,
+            resolve_result_storage,
+        )
         from prefect.states import Pending, Scheduled
         from prefect.tasks import Task
 
@@ -2563,26 +2681,36 @@ class InfrastructureBoundFlow(Flow[P, R]):
                 )
 
             current_result_store = get_result_store()
-            # Check result storage and use the work pool default if needed
-            if self.result_storage is None and (
-                current_result_store.result_storage is None
-                or isinstance(current_result_store.result_storage, LocalFileSystem)
+            if not _result_storage_is_configured_for_remote_retrieval(
+                self.result_storage,
+                current_result_store.result_storage,
             ):
+                result_storage = None
                 if (
                     work_pool.storage_configuration.default_result_storage_block_id
-                    is None
+                    is not None
                 ):
+                    result_storage = resolve_result_storage(
+                        work_pool.storage_configuration.default_result_storage_block_id,
+                        _sync=True,
+                    )
+                else:
+                    default_result_storage = _get_default_result_storage()
+                    if (
+                        default_result_storage.source
+                        is not _DefaultResultStorageSource.LOCAL_STORAGE_PATH
+                    ):
+                        result_storage = default_result_storage.storage
+
+                if result_storage is None:
                     logger.warning(
                         f"Flow {self.name!r} has no result storage configured. Please configure "
                         "result storage for the flow if you want to retrieve the result for the flow run."
                     )
+                    flow = self
                 else:
-                    # Use the work pool's default result storage block for the flow run to ensure the caller can retrieve the result
                     flow = self.with_options(
-                        result_storage=resolve_result_storage(
-                            work_pool.storage_configuration.default_result_storage_block_id,
-                            _sync=True,
-                        ),
+                        result_storage=result_storage,
                         persist_result=True,
                     )
             else: