dex-python-sdk 1.12.0__tar.gz → 1.14.1__tar.gz

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dex-python-sdk
-Version: 1.12.0
+Version: 1.14.1
 Requires-Dist: grpcio>=1.83.0
 Requires-Dist: grpcio-status>=1.83.0
 Requires-Dist: protobuf>=7.35.1
@@ -94,11 +94,13 @@ The Worker polls its sticky queue and every registered Flow type queue. Flow
 type names must be unique within a Dex server. `max_concurrent_tasks` defaults
 to the CPU count, and `idle_streams_per_queue` defaults to one. At capacity,
 the Worker rejects an assignment so Dex can reassign it. Each stream runs one
-task at a time; Step heartbeat and stream writes use the same stream. A Worker
-may hold up to `(1 + number of Flow types) * (idle_streams_per_queue +
-max_concurrent_tasks)` streams because busy streams are replaced with idle
-streams and retained after tasks finish. Size these options for the server's
-per-Worker stream cap (256 by default).
+task at a time; Step heartbeat and stream writes use the same stream. After tasks finish, the Worker
+gracefully closes surplus idle streams while preserving readiness. After load drops,
+the steady-state bound is `max_concurrent_tasks + (1 + number of Flow types) *
+idle_streams_per_queue` streams. The transient peak during a burst remains bounded
+by the matching caps. Size these options for `matching.maxConcurrentTasksCap` per
+app Worker ID per matching owner (256 by default) and `matching.maxStreams` per
+Matching instance (100,000 by default).
 
 The Worker rejects an unregistered Step or RPC by name. Dex retries that task
 on another pull Worker without consuming a Step attempt, including when the
@@ -122,7 +124,7 @@ an active task to finish, then waits up to `stop_drain_timeout` for the server
 to close the stream after the final frame. An idle stream reconnects
 immediately. An older server without pull support produces one explicit error
 log and is retried after one minute. `start()` reports startup
-synchronization errors, while later stream errors are logged and retried.
+synchronization errors, while later stream errors are logged and retried by default.
 `stop()` drains for at most
 `stop_drain_timeout`, which defaults to 30 seconds. Afterward it requests
 cancellation and waits for handlers to return. Synchronous handlers should
@@ -130,6 +132,31 @@ check `Context.is_cancellation_requested()` during long operations. Streams
 that sent a final frame wait for server EOF until the remaining Stop drain
 deadline before the channel closes.
 
+Call `worker.wait_pull_registered(timeout=30)` after starting a synchronous
+Worker on another thread. For AsyncWorker, start it in another task and use
+`await worker.wait_pull_registered(timeout=30)`. The timeout is in seconds;
+`None` waits indefinitely. Each waits for every Flow type queue and the app
+Worker ID queue to register at least once. Timeout raises `TimeoutError`;
+async cancellation cancels only that wait. Startup failures reach waiters.
+A permanent rejection before readiness raises `PullRegistrationError`, whose
+`queue` and `error` identify the rejected queue and original transport error.
+Listening mode and stop before readiness raise `RuntimeError`.
+
+Use `worker.pull_registration_state()` for subsequent readiness probes. Its
+immutable `queues` expose `queue`, `has_registered_stream`, and
+`last_permanent_error`. After initial registration, `is_ready` stays true across
+transient failures, reconnects, and stop. It becomes false only when a queue
+has a permanent error and no registered stream. Re-registration clears the
+error. The startup wait keeps succeeding after initial registration, including
+after stop. Multiple callers may wait independently before or after start.
+
+`PullWorkerOptions(stop_on_permanent_stream_rejection=True)` makes the first
+permanent rejection leaving a queue without a registered stream drain within
+`stop_drain_timeout`, then raise `PullRegistrationError` from `start()`. This
+applies before and after readiness. Another registered stream for that queue
+prevents shutdown. The default is False and retries indefinitely. Always call
+`stop()` or `close()` afterward; both remain safe after `start()` returns.
+
 ## Flow state I/O
 
 Applications read and write Flow state through typed RPCs. Client and AsyncClient

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/README.md

@@ -78,11 +78,13 @@ The Worker polls its sticky queue and every registered Flow type queue. Flow
 type names must be unique within a Dex server. `max_concurrent_tasks` defaults
 to the CPU count, and `idle_streams_per_queue` defaults to one. At capacity,
 the Worker rejects an assignment so Dex can reassign it. Each stream runs one
-task at a time; Step heartbeat and stream writes use the same stream. A Worker
-may hold up to `(1 + number of Flow types) * (idle_streams_per_queue +
-max_concurrent_tasks)` streams because busy streams are replaced with idle
-streams and retained after tasks finish. Size these options for the server's
-per-Worker stream cap (256 by default).
+task at a time; Step heartbeat and stream writes use the same stream. After tasks finish, the Worker
+gracefully closes surplus idle streams while preserving readiness. After load drops,
+the steady-state bound is `max_concurrent_tasks + (1 + number of Flow types) *
+idle_streams_per_queue` streams. The transient peak during a burst remains bounded
+by the matching caps. Size these options for `matching.maxConcurrentTasksCap` per
+app Worker ID per matching owner (256 by default) and `matching.maxStreams` per
+Matching instance (100,000 by default).
 
 The Worker rejects an unregistered Step or RPC by name. Dex retries that task
 on another pull Worker without consuming a Step attempt, including when the
@@ -106,7 +108,7 @@ an active task to finish, then waits up to `stop_drain_timeout` for the server
 to close the stream after the final frame. An idle stream reconnects
 immediately. An older server without pull support produces one explicit error
 log and is retried after one minute. `start()` reports startup
-synchronization errors, while later stream errors are logged and retried.
+synchronization errors, while later stream errors are logged and retried by default.
 `stop()` drains for at most
 `stop_drain_timeout`, which defaults to 30 seconds. Afterward it requests
 cancellation and waits for handlers to return. Synchronous handlers should
@@ -114,6 +116,31 @@ check `Context.is_cancellation_requested()` during long operations. Streams
 that sent a final frame wait for server EOF until the remaining Stop drain
 deadline before the channel closes.
 
+Call `worker.wait_pull_registered(timeout=30)` after starting a synchronous
+Worker on another thread. For AsyncWorker, start it in another task and use
+`await worker.wait_pull_registered(timeout=30)`. The timeout is in seconds;
+`None` waits indefinitely. Each waits for every Flow type queue and the app
+Worker ID queue to register at least once. Timeout raises `TimeoutError`;
+async cancellation cancels only that wait. Startup failures reach waiters.
+A permanent rejection before readiness raises `PullRegistrationError`, whose
+`queue` and `error` identify the rejected queue and original transport error.
+Listening mode and stop before readiness raise `RuntimeError`.
+
+Use `worker.pull_registration_state()` for subsequent readiness probes. Its
+immutable `queues` expose `queue`, `has_registered_stream`, and
+`last_permanent_error`. After initial registration, `is_ready` stays true across
+transient failures, reconnects, and stop. It becomes false only when a queue
+has a permanent error and no registered stream. Re-registration clears the
+error. The startup wait keeps succeeding after initial registration, including
+after stop. Multiple callers may wait independently before or after start.
+
+`PullWorkerOptions(stop_on_permanent_stream_rejection=True)` makes the first
+permanent rejection leaving a queue without a registered stream drain within
+`stop_drain_timeout`, then raise `PullRegistrationError` from `start()`. This
+applies before and after readiness. Another registered stream for that queue
+prevents shutdown. The default is False and retries indefinitely. Always call
+`stop()` or `close()` afterward; both remain safe after `start()` returns.
+
 ## Flow state I/O
 
 Applications read and write Flow state through typed RPCs. Client and AsyncClient

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/dex/__init__.py

@@ -113,6 +113,11 @@ from dex.timer import Timer
 from dex.wait import Wait
 from dex.wait_options import WaitForAttributeOptions, WaitForStepCompletionOptions
 from dex.worker import Worker
+from dex.pull_registration import (
+    PullRegistrationError,
+    PullRegistrationState,
+    PullQueueRegistrationState,
+)
 from dex.worker_options import PullWorkerOptions, WorkerOptions, WorkerTarget
 
 __all__ = [
@@ -214,6 +219,9 @@ __all__ = [
     "WorkerOptions",
     "WorkerTarget",
     "PullWorkerOptions",
+    "PullRegistrationError",
+    "PullRegistrationState",
+    "PullQueueRegistrationState",
     "WaitForAttributeOptions",
     "WaitForStepCompletionOptions",
     "dead_end",

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/dex/_pull_worker.py

@@ -28,6 +28,7 @@ from dex._server_protocol import sdk_version
 from dex.dexpb import dex_pb2 as pb
 from dex.dexpb import dex_pb2_grpc
 from dex.flow import Registry
+from dex.pull_registration import PullRegistrationError, _PullRegistration
 from dex.worker_options import PullWorkerOptions
 
 _LOGGER = logging.getLogger(__name__)
@@ -86,6 +87,12 @@ class PullWorkerRuntime:
         self.app_worker_id = app_worker_id or resolve_app_worker_id(registry, options)
         self.flow_types = sorted(registry._registered_flows)
         self.queues = (self.app_worker_id, *self.flow_types)
+        self.registration = _PullRegistration(self.queues)
+        self._stop_on_permanent_stream_rejection = (
+            options.stop_on_permanent_stream_rejection
+        )
+        self._permanent_error: PullRegistrationError | None = None
+        self._rejection_stop_task: asyncio.Task[None] | None = None
         self.max_concurrent_tasks = options.max_concurrent_tasks or os.cpu_count() or 1
         self.idle_streams_per_queue = options.idle_streams_per_queue or 1
         self.stop_drain_timeout = options.stop_drain_timeout.total_seconds() or 30
@@ -131,9 +138,10 @@ class PullWorkerRuntime:
         self._stopping = asyncio.Event()
         self._stop_completed = asyncio.Event()
         self._started.set()
-        channel = outbound_worker_task_channel(self._server_address)
-        stub = dex_pb2_grpc.WorkerTaskServiceStub(channel)  # type: ignore[no-untyped-call]
+        channel: grpc.aio.Channel | None = None
         try:
+            channel = outbound_worker_task_channel(self._server_address)
+            stub = dex_pb2_grpc.WorkerTaskServiceStub(channel)  # type: ignore[no-untyped-call]
             self._wake.set()
             while not self._stopping.is_set():
                 await self._wake.wait()
@@ -144,9 +152,14 @@ class PullWorkerRuntime:
                 *(stream.run_task for stream in tuple(self._streams)),
                 return_exceptions=True,
             )
+            if self._rejection_stop_task is not None:
+                await self._rejection_stop_task
+            if self._permanent_error is not None:
+                raise self._permanent_error
         finally:
             try:
-                await channel.close(grace=None)
+                if channel is not None:
+                    await channel.close(grace=None)
             finally:
                 self._loop_thread_id = None
 
@@ -157,6 +170,8 @@ class PullWorkerRuntime:
     def _reconcile(self, stub: Any) -> None:
         if self._stopping is None or self._stopping.is_set():
             return
+        for stream in tuple(self._streams):
+            stream.try_retire()
         for queue in self.queues:
             idle = sum(
                 stream.queue == queue
@@ -192,6 +207,7 @@ class PullWorkerRuntime:
         self.signal()
 
     async def stop(self) -> None:
+        self.registration.record_stopping()
         if self._stopping is None:
             return
         if self._stopping.is_set():
@@ -263,7 +279,10 @@ class PullStream:
         self.handler_task: asyncio.Task[None] | None = None
         self.task_id = ""
         self.is_busy = False
+        self._is_registered = False
         self.is_draining = False
+        self._is_retiring = False
+        self._retirement_requested = asyncio.Event()
         self.is_half_closing = False
         self._active = threading.Event()
         self._call: Any = None
@@ -289,7 +308,28 @@ class PullStream:
                 _LOGGER.exception(
                     "Worker task stream failed", extra={"queue": self.queue}
                 )
-            if self.runtime.is_stopping:
+            finally:
+                code = error.code() if error is not None else None
+                rejection = self.runtime.registration.record_stream_result(
+                    self.queue,
+                    self._is_registered,
+                    error if code in _PERMANENT_CODES else None,
+                )
+                self._is_registered = False
+                if (
+                    rejection is not None
+                    and self.runtime._stop_on_permanent_stream_rejection
+                    and self.runtime._permanent_error is None
+                ):
+                    self.runtime._permanent_error = rejection
+                    self.runtime._rejection_stop_task = asyncio.create_task(
+                        self.runtime.stop()
+                    )
+            if (
+                self.runtime.is_stopping
+                or self.runtime._permanent_error is not None
+                or self._is_retiring
+            ):
                 break
             if self._is_reconnecting:
                 self._is_reconnecting = False
@@ -346,11 +386,12 @@ class PullStream:
         )
         was_registered = False
         reader: asyncio.Task[Any] | None = None
+        retirement = asyncio.create_task(self._retirement_requested.wait())
         try:
             while not self.runtime.is_stopping or self.is_busy:
                 if reader is None:
                     reader = asyncio.create_task(self._call.read())
-                waiting: set[asyncio.Task[Any]] = {reader}
+                waiting: set[asyncio.Task[Any]] = {reader, retirement}
                 if self.handler_task is not None:
                     waiting.add(self.handler_task)
                 completed, _ = await asyncio.wait(
@@ -358,22 +399,23 @@ class PullStream:
                 )
                 if self.handler_task in completed:
                     await self._finish_handler()
-                    if self.is_draining:
-                        if reader.done():
-                            frame = reader.result()
-                            reader = None
-                            if (
-                                frame is not _GRPC_AIO_EOF
-                                and frame.WhichOneof("frame") == "task"
-                            ):
-                                await self._reject_task(frame.task)
-                        self.is_half_closing = True
-                        self.runtime.signal()
-                        await self._call.done_writing()
-                        await self._await_stream_eof(reader)
+                    self.try_retire()
+                if self.is_draining and not self.is_busy:
+                    if reader.done():
+                        frame = reader.result()
                         reader = None
-                        self._is_reconnecting = True
-                        return was_registered
+                        if (
+                            frame is not _GRPC_AIO_EOF
+                            and frame.WhichOneof("frame") == "task"
+                        ):
+                            await self._reject_task(frame.task)
+                    self.is_half_closing = True
+                    self.runtime.signal()
+                    await self._call.done_writing()
+                    await self._await_stream_eof(reader)
+                    reader = None
+                    self._is_reconnecting = not self._is_retiring
+                    return was_registered
                 if reader not in completed:
                     continue
                 frame = reader.result()
@@ -391,6 +433,9 @@ class PullStream:
                         frame.registered.supports_unknown_name_rejection
                     )
                     was_registered = True
+                    self._is_registered = True
+                    self.runtime.registration.record_registered_stream(self.queue)
+                    self.runtime.signal()
                     continue
                 if kind == "task":
                     await self._start_task(frame.task)
@@ -408,6 +453,8 @@ class PullStream:
                         self._is_reconnecting = True
                         return was_registered
         finally:
+            retirement.cancel()
+            await asyncio.gather(retirement, return_exceptions=True)
             self.is_half_closing = False
             self.cancel_handler()
             if self.handler_task is not None:
@@ -624,6 +671,22 @@ class PullStream:
         self._active.clear()
         self.runtime.finish_task()
 
+    def try_retire(self) -> None:
+        if self.is_busy or self.is_draining or not self._is_registered:
+            return
+        idle_replacements = sum(
+            candidate is not self
+            and candidate.queue == self.queue
+            and candidate._is_registered
+            and not candidate.is_busy
+            and not candidate.is_draining
+            for candidate in self.runtime._streams
+        )
+        if idle_replacements >= self.runtime.idle_streams_per_queue:
+            self.is_draining = True
+            self._is_retiring = True
+            self._retirement_requested.set()
+
     def cancel_handler(self) -> None:
         self._active.clear()
         if self.runtime._executor is None and self.handler_task is not None:

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/dex/async_worker.py

@@ -30,6 +30,7 @@ from dex.blob_cache import BlobCache
 from dex.dexpb import dex_pb2 as pb
 from dex.dexpb import dex_pb2_grpc
 from dex.flow import Registry
+from dex.pull_registration import PullRegistrationState
 from dex.worker_options import WorkerOptions, WorkerTarget
 
 
@@ -100,6 +101,7 @@ class AsyncWorker:
             self._server_protocol,
         )
         self._server = None
+        self._has_start_called = False
         self._pull = None
         if self.options.pull is not None:
             self._pull = PullWorkerRuntime(
@@ -167,11 +169,77 @@ class AsyncWorker:
 
         ``start`` may be awaited exactly once. It contacts FlowService before binding
         and completes only after ``stop`` terminates the listener or pull streams.
-        Later pull stream failures are logged and retried.
+        Permanent pull stream rejections raise PullRegistrationError when
+        stop_on_permanent_stream_rejection is enabled; otherwise streams retry.
 
         Raises:
+            PullRegistrationError: A permanent stream rejection stops this Worker.
             RuntimeError: If lifecycle state, index synchronization, or binding fails.
         """
+        if self._has_start_called:
+            raise RuntimeError("Worker can only start once")
+        self._has_start_called = True
+        start_error: BaseException | None = None
+        try:
+            await self._start()
+        except BaseException as failure:
+            start_error = failure
+            raise
+        finally:
+            if self._pull is not None:
+                self._pull.registration.finish_start(start_error)
+                if self._state == "running":
+                    await self.stop()
+
+    async def wait_pull_registered(self, timeout: float | None = None) -> None:
+        """Wait until every pull queue has registered at least once.
+
+        Start the Worker separately before waiting, or start it concurrently.
+        Successful registration latches, including after stop. Use
+        pull_registration_state for subsequent readiness probes. A permanent
+        rejection before readiness raises PullRegistrationError only when no
+        other stream is registered for that queue. Startup failures wake waiters.
+        Multiple callers may wait independently, including before start.
+
+        Args:
+            timeout: Maximum seconds to wait; None waits indefinitely.
+
+        Raises:
+            PullRegistrationError: A queue is permanently rejected before readiness.
+            TimeoutError: The timeout expires.
+            asyncio.CancelledError: The calling task is canceled; the Worker keeps running.
+            RuntimeError: This is a listening Worker, or it stops before readiness.
+            BaseException: The original startup failure, if start fails.
+
+        Examples:
+            >>> start_task = asyncio.create_task(worker.start())
+
+            >>> await worker.wait_pull_registered(timeout=30)
+            >>> is_ready = worker.pull_registration_state().is_ready
+        """
+        if self._pull is None:
+            raise RuntimeError("wait_pull_registered requires a pull Worker")
+        await self._pull.registration.wait_async(timeout)
+
+    def pull_registration_state(self) -> PullRegistrationState:
+        """Return an immutable snapshot of pull queue registrations.
+
+        Transient failures, reconnects, and stop preserve initial readiness.
+        A permanent rejection makes readiness false only while that queue has no
+        registered stream. Re-registration clears its error. Calls are safe
+        concurrently with start, stop, and other waiters. See wait_pull_registered
+        for an example.
+
+        Returns:
+            Current per-queue state, or an empty nonready state in listening mode.
+        """
+        return (
+            PullRegistrationState()
+            if self._pull is None
+            else self._pull.registration.snapshot()
+        )
+
+    async def _start(self) -> None:
         if self._state != "created":
             raise RuntimeError(f"AsyncWorker cannot start from state {self._state}")
         try:
@@ -247,6 +315,8 @@ class AsyncWorker:
         Half-closed pull streams wait for server EOF until the remaining drain
         deadline before the channel closes.
         """
+        if self._pull is not None:
+            self._pull.registration.record_stopping()
         if self._state in ("stopped", "closed"):
             return
         self._state = "stopping"

dex_python_sdk-1.14.1/dex/pull_registration.py

@@ -0,0 +1,181 @@
+# Copyright (c) 2026 Super Durable, Inc.
+#
+# Licensed under the Sustainable Use License 1.0.
+# You may not use this file except in compliance with the License.
+# See the LICENSE file in the repository root.
+#
+# SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0
+
+import asyncio
+import threading
+import time
+from concurrent.futures import Future
+from dataclasses import dataclass
+
+
+class PullRegistrationError(RuntimeError):
+    """Identify a permanent rejection leaving a pull queue without a registered stream.
+
+    Raised by ``wait_pull_registered`` before initial readiness, or by ``start``
+    when ``stop_on_permanent_stream_rejection`` is enabled. Inspect ``queue``
+    and ``error`` for the rejected queue and original gRPC error. Permanent codes
+    are INVALID_ARGUMENT, RESOURCE_EXHAUSTED, UNIMPLEMENTED, PERMISSION_DENIED,
+    and UNAUTHENTICATED.
+
+    Attributes:
+        queue: Rejected Flow type or app Worker ID queue.
+        error: Original permanent transport error, also exposed as ``__cause__``.
+    """
+
+    def __init__(self, queue: str, error: BaseException) -> None:
+        """Create a queue-specific rejection.
+
+        Args:
+            queue: Rejected queue name.
+            error: Original permanent transport error.
+        """
+        super().__init__(f"pull Worker queue {queue!r} registration rejected: {error}")
+        self.queue = queue
+        self.error = error
+        self.__cause__ = error
+
+
+@dataclass(frozen=True)
+class PullQueueRegistrationState:
+    """Describe one queue in an immutable readiness snapshot.
+
+    Attributes:
+        queue: Flow type or app Worker ID queue.
+        has_registered_stream: Whether a stream is currently registered.
+        last_permanent_error: Last permanent transport error, or ``None``.
+            A new Registered acknowledgement clears it. Treat errors as immutable.
+    """
+
+    queue: str
+    has_registered_stream: bool
+    last_permanent_error: BaseException | None
+
+
+@dataclass(frozen=True)
+class PullRegistrationState:
+    """Describe pull registration readiness independently of the Worker lifecycle.
+
+    Obtain snapshots with ``worker.pull_registration_state()``. After every queue
+    has registered once, transient failures, reconnects, and stop keep readiness.
+    A permanent error with no registered stream makes ``is_ready`` false until
+    re-registration. A listening Worker returns ``False`` and an empty tuple.
+
+    Attributes:
+        is_ready: Whether all queues registered once and no queue is permanently
+            rejected without a registered stream.
+        queues: Immutable per-queue snapshots, ordered by app Worker ID then Flow type.
+
+    Examples:
+        >>> state = worker.pull_registration_state()
+        >>> is_ready = state.is_ready
+    """
+
+    is_ready: bool = False
+    queues: tuple[PullQueueRegistrationState, ...] = ()
+
+
+class _PullRegistration:
+    def __init__(self, queues: tuple[str, ...]) -> None:
+        self._lock = threading.Lock()
+        self._queues = queues
+        self._counts = dict.fromkeys(queues, 0)
+        self._registered: set[str] = set()
+        self._errors: dict[str, BaseException] = {}
+        self._initial_error: PullRegistrationError | None = None
+        self._has_registered_all = False
+        self._is_stopping = False
+        self._has_start_returned = False
+        self._start_error: BaseException | None = None
+        self._changed: Future[None] = Future()
+
+    def record_registered_stream(self, queue: str) -> None:
+        with self._lock:
+            self._counts[queue] += 1
+            self._registered.add(queue)
+            self._errors.pop(queue, None)
+            self._has_registered_all = len(self._registered) == len(self._queues)
+            self._notify()
+
+    def record_stream_result(
+        self, queue: str, was_registered: bool, error: BaseException | None
+    ) -> PullRegistrationError | None:
+        with self._lock:
+            if was_registered:
+                self._counts[queue] -= 1
+            rejection = None
+            if error is not None:
+                self._errors[queue] = error
+                if self._counts[queue] == 0:
+                    rejection = PullRegistrationError(queue, error)
+                    if not self._has_registered_all and self._initial_error is None:
+                        self._initial_error = rejection
+            self._notify()
+            return rejection
+
+    def record_stopping(self) -> None:
+        with self._lock:
+            self._is_stopping = True
+            self._notify()
+
+    def finish_start(self, error: BaseException | None) -> None:
+        with self._lock:
+            self._has_start_returned = True
+            self._start_error = error
+            self._notify()
+
+    def snapshot(self) -> PullRegistrationState:
+        with self._lock:
+            queues = tuple(
+                PullQueueRegistrationState(
+                    queue, self._counts[queue] > 0, self._errors.get(queue)
+                )
+                for queue in self._queues
+            )
+            return PullRegistrationState(
+                self._has_registered_all
+                and not any(
+                    not queue.has_registered_stream
+                    and queue.last_permanent_error is not None
+                    for queue in queues
+                ),
+                queues,
+            )
+
+    def wait(self, timeout: float | None) -> None:
+        deadline = None if timeout is None else time.monotonic() + timeout
+        while (changed := self._next_change()) is not None:
+            changed.result(
+                None if deadline is None else max(0, deadline - time.monotonic())
+            )
+
+    async def wait_async(self, timeout: float | None) -> None:
+        if self._next_change() is not None:
+            await asyncio.wait_for(self._wait_async(), timeout)
+
+    async def _wait_async(self) -> None:
+        while (changed := self._next_change()) is not None:
+            await asyncio.shield(asyncio.wrap_future(changed))
+
+    def _next_change(self) -> Future[None] | None:
+        with self._lock:
+            if self._has_registered_all:
+                return None
+            if self._initial_error is not None:
+                raise self._initial_error
+            if self._has_start_returned:
+                if self._start_error is not None:
+                    raise self._start_error
+                raise RuntimeError("pull Worker stopped before registration")
+            if self._is_stopping:
+                raise RuntimeError("pull Worker stopped before registration")
+            return self._changed
+
+    def _notify(self) -> None:
+        changed = self._changed
+        self._changed = Future()
+        changed.set_result(None)

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/dex/worker.py

@@ -35,6 +35,7 @@ from dex.blob_cache import BlobCache
 from dex.dexpb import dex_pb2 as pb
 from dex.dexpb import dex_pb2_grpc
 from dex.flow import Registry
+from dex.pull_registration import PullRegistrationState
 from dex.worker_options import WorkerOptions, WorkerTarget
 
 
@@ -117,6 +118,7 @@ class Worker:
             thread_name_prefix="dex-python-handler",
         )
         self._server = None
+        self._has_start_called = False
         self._pull = None
         if self.options.pull is not None:
             self._pull = PullWorkerRuntime(
@@ -188,12 +190,78 @@ class Worker:
 
         ``start`` may be called exactly once. It first contacts FlowService using the
         configured synchronization timeout, then starts a listener or outbound
-        pull streams and waits until ``stop`` finishes. Later stream failures are
-        logged and retried; they do not raise from ``start``.
+        pull streams and waits until ``stop`` finishes. Permanent stream rejections
+        raise PullRegistrationError when stop_on_permanent_stream_rejection is
+        enabled; otherwise streams retry.
 
         Raises:
+            PullRegistrationError: A permanent stream rejection stops this Worker.
             RuntimeError: If lifecycle state, index synchronization, or binding fails.
         """
+        with self._lock:
+            if self._has_start_called or self._state != "created":
+                raise RuntimeError(f"Worker cannot start from state {self._state}")
+            self._has_start_called = True
+        start_error: BaseException | None = None
+        try:
+            self._start()
+        except BaseException as failure:
+            start_error = failure
+            raise
+        finally:
+            if self._pull is not None:
+                self._pull.registration.finish_start(start_error)
+                if self._state == "running":
+                    self.stop()
+
+    def wait_pull_registered(self, timeout: float | None = None) -> None:
+        """Wait until every pull queue has registered at least once.
+
+        Start the Worker separately before waiting, or start it concurrently.
+        Successful registration latches, including after stop. Use
+        pull_registration_state for subsequent readiness probes. A permanent
+        rejection before readiness raises PullRegistrationError only when no
+        other stream is registered for that queue. Startup failures wake waiters.
+        Multiple callers may wait independently, including before start.
+
+        Args:
+            timeout: Maximum seconds to wait; None waits indefinitely.
+
+        Raises:
+            PullRegistrationError: A queue is permanently rejected before readiness.
+            TimeoutError: The timeout expires.
+            RuntimeError: This is a listening Worker, or it stops before readiness.
+            BaseException: The original startup failure, if start fails.
+
+        Examples:
+            >>> start_thread = threading.Thread(target=worker.start)
+            >>> start_thread.start()
+            >>> worker.wait_pull_registered(timeout=30)
+            >>> is_ready = worker.pull_registration_state().is_ready
+        """
+        if self._pull is None:
+            raise RuntimeError("wait_pull_registered requires a pull Worker")
+        self._pull.registration.wait(timeout)
+
+    def pull_registration_state(self) -> PullRegistrationState:
+        """Return an immutable snapshot of pull queue registrations.
+
+        Transient failures, reconnects, and stop preserve initial readiness.
+        A permanent rejection makes readiness false only while that queue has no
+        registered stream. Re-registration clears its error. Calls are safe
+        concurrently with start, stop, and other waiters. See wait_pull_registered
+        for an example.
+
+        Returns:
+            Current per-queue state, or an empty nonready state in listening mode.
+        """
+        return (
+            PullRegistrationState()
+            if self._pull is None
+            else self._pull.registration.snapshot()
+        )
+
+    def _start(self) -> None:
         with self._lock:
             if self._state != "created":
                 raise RuntimeError(f"Worker cannot start from state {self._state}")
@@ -291,6 +359,7 @@ class Worker:
             was_created = self._state == "created"
             self._state = "stopping"
         if self._pull is not None:
+            self._pull.registration.record_stopping()
             if not was_created:
                 self._pull.stop_threadsafe()
         else:

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/dex/worker_options.py

@@ -77,6 +77,11 @@ class PullWorkerOptions:
             the final frame. During Stop, half-closed streams use the remaining
             drain deadline.
 
+        stop_on_permanent_stream_rejection: Whether the first permanent rejection
+            leaving a queue without a registered stream drains the Worker and
+            raises PullRegistrationError from start. Defaults to False, which
+            retries indefinitely. Applies before and after initial readiness.
+
     Raises:
         ValueError: If a numeric option is negative or app_worker_id is invalid.
 
@@ -91,6 +96,7 @@ class PullWorkerOptions:
     idle_streams_per_queue: int = 0
     app_worker_id: str = ""
     stop_drain_timeout: timedelta = timedelta(0)
+    stop_on_permanent_stream_rejection: bool = False
 
     def __post_init__(self) -> None:
         if self.max_concurrent_tasks < 0:

{dex_python_sdk-1.12.0 → dex_python_sdk-1.14.1}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "dex-python-sdk"
 # Publish uses tag sdk-python/v*; CI stamps this for release builds.
-version = "1.12.0"
+version = "1.14.1"
 description = "Python SDK for the Dex workflow engine"
 authors = [{name = "Super Durable"}]
 readme = "README.md"