dex-python-sdk 1.14.0__tar.gz → 1.15.0__tar.gz

{dex_python_sdk-1.14.0 → dex_python_sdk-1.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dex-python-sdk
-Version: 1.14.0
+Version: 1.15.0
 Requires-Dist: grpcio>=1.83.0
 Requires-Dist: grpcio-status>=1.83.0
 Requires-Dist: protobuf>=7.35.1
@@ -94,11 +94,13 @@ The Worker polls its sticky queue and every registered Flow type queue. Flow
 type names must be unique within a Dex server. `max_concurrent_tasks` defaults
 to the CPU count, and `idle_streams_per_queue` defaults to one. At capacity,
 the Worker rejects an assignment so Dex can reassign it. Each stream runs one
-task at a time; Step heartbeat and stream writes use the same stream. A Worker
-may hold up to `(1 + number of Flow types) * (idle_streams_per_queue +
-max_concurrent_tasks)` streams because busy streams are replaced with idle
-streams and retained after tasks finish. Size these options for the server's
-per-Worker stream cap (256 by default).
+task at a time; Step heartbeat and stream writes use the same stream. After tasks finish, the Worker
+gracefully closes surplus idle streams while preserving readiness. After load drops,
+the steady-state bound is `max_concurrent_tasks + (1 + number of Flow types) *
+idle_streams_per_queue` streams. The transient peak during a burst remains bounded
+by the matching caps. Size these options for `matching.maxConcurrentTasksCap` per
+app Worker ID per matching owner (256 by default) and `matching.maxStreams` per
+Matching instance (100,000 by default).
 
 The Worker rejects an unregistered Step or RPC by name. Dex retries that task
 on another pull Worker without consuming a Step attempt, including when the
@@ -155,6 +157,40 @@ applies before and after readiness. Another registered stream for that queue
 prevents shutdown. The default is False and retries indefinitely. Always call
 `stop()` or `close()` afterward; both remain safe after `start()` returns.
 
+### Reserved RPC capacity
+
+Set `reserved_rpc_tasks` when synchronous RPC latency must stay responsive
+behind long-running Steps. When unset, it reserves one slot if the resolved task
+limit is at least two, otherwise zero. With the CPU-count default, Steps get one
+fewer slot than before. Set it explicitly to zero to opt out. With one slot and
+no explicit reserve, Worker start logs one warning: RPCs may wait behind
+long-running Steps; use a limit of at least two or set the reserve explicitly.
+Flows must stay responsive; Dex cannot prioritize synchronous RPCs like workflow
+engines prioritize reads. The value must be nonnegative and less than
+`max_concurrent_tasks` after its CPU-count default is resolved; invalid values
+fail construction. RPC and Step handlers count together: WaitFor and Execute
+tasks, including local attempts, are declined with `NO_CAPACITY` when the busy
+count reaches the task limit minus the reserve. API-direct and transactional
+InvokeWorkerRPC tasks can use the full limit.
+
+```python
+pull = PullWorkerOptions(max_concurrent_tasks=8, reserved_rpc_tasks=2)
+```
+
+A decline lets Dex try another Worker without consuming an attempt. If all
+Workers remain at their Step limit, a Step can wait until the dispatch wait
+expires, then receive `NO_WORKER_AVAILABLE`, consume an attempt, and follow its
+retry policy, as when all Workers are full. Local attempts wait the local
+dispatch wait, then fall back according to their policy. Each queue still keeps
+its configured idle streams; a reserve decline does not close a stream or change
+registration readiness.
+
+Each reserve decline records the Flow type through the dex._pull_worker logger
+at DEBUG level. Export the cumulative `pull_reserved_rpc_declines` Worker value
+to distinguish reserve pressure from full-capacity rejections. The count
+excludes full-capacity rejections and unknown names, never resets, and remains
+available after shutdown. Listening Workers return zero.
+
 ## Flow state I/O
 
 Applications read and write Flow state through typed RPCs. Client and AsyncClient

{dex_python_sdk-1.14.0 → dex_python_sdk-1.15.0}/README.md

@@ -78,11 +78,13 @@ The Worker polls its sticky queue and every registered Flow type queue. Flow
 type names must be unique within a Dex server. `max_concurrent_tasks` defaults
 to the CPU count, and `idle_streams_per_queue` defaults to one. At capacity,
 the Worker rejects an assignment so Dex can reassign it. Each stream runs one
-task at a time; Step heartbeat and stream writes use the same stream. A Worker
-may hold up to `(1 + number of Flow types) * (idle_streams_per_queue +
-max_concurrent_tasks)` streams because busy streams are replaced with idle
-streams and retained after tasks finish. Size these options for the server's
-per-Worker stream cap (256 by default).
+task at a time; Step heartbeat and stream writes use the same stream. After tasks finish, the Worker
+gracefully closes surplus idle streams while preserving readiness. After load drops,
+the steady-state bound is `max_concurrent_tasks + (1 + number of Flow types) *
+idle_streams_per_queue` streams. The transient peak during a burst remains bounded
+by the matching caps. Size these options for `matching.maxConcurrentTasksCap` per
+app Worker ID per matching owner (256 by default) and `matching.maxStreams` per
+Matching instance (100,000 by default).
 
 The Worker rejects an unregistered Step or RPC by name. Dex retries that task
 on another pull Worker without consuming a Step attempt, including when the
@@ -139,6 +141,40 @@ applies before and after readiness. Another registered stream for that queue
 prevents shutdown. The default is False and retries indefinitely. Always call
 `stop()` or `close()` afterward; both remain safe after `start()` returns.
 
+### Reserved RPC capacity
+
+Set `reserved_rpc_tasks` when synchronous RPC latency must stay responsive
+behind long-running Steps. When unset, it reserves one slot if the resolved task
+limit is at least two, otherwise zero. With the CPU-count default, Steps get one
+fewer slot than before. Set it explicitly to zero to opt out. With one slot and
+no explicit reserve, Worker start logs one warning: RPCs may wait behind
+long-running Steps; use a limit of at least two or set the reserve explicitly.
+Flows must stay responsive; Dex cannot prioritize synchronous RPCs like workflow
+engines prioritize reads. The value must be nonnegative and less than
+`max_concurrent_tasks` after its CPU-count default is resolved; invalid values
+fail construction. RPC and Step handlers count together: WaitFor and Execute
+tasks, including local attempts, are declined with `NO_CAPACITY` when the busy
+count reaches the task limit minus the reserve. API-direct and transactional
+InvokeWorkerRPC tasks can use the full limit.
+
+```python
+pull = PullWorkerOptions(max_concurrent_tasks=8, reserved_rpc_tasks=2)
+```
+
+A decline lets Dex try another Worker without consuming an attempt. If all
+Workers remain at their Step limit, a Step can wait until the dispatch wait
+expires, then receive `NO_WORKER_AVAILABLE`, consume an attempt, and follow its
+retry policy, as when all Workers are full. Local attempts wait the local
+dispatch wait, then fall back according to their policy. Each queue still keeps
+its configured idle streams; a reserve decline does not close a stream or change
+registration readiness.
+
+Each reserve decline records the Flow type through the dex._pull_worker logger
+at DEBUG level. Export the cumulative `pull_reserved_rpc_declines` Worker value
+to distinguish reserve pressure from full-capacity rejections. The count
+excludes full-capacity rejections and unknown names, never resets, and remains
+available after shutdown. Listening Workers return zero.
+
 ## Flow state I/O
 
 Applications read and write Flow state through typed RPCs. Client and AsyncClient

{dex_python_sdk-1.14.0 → dex_python_sdk-1.15.0}/dex/_pull_worker.py

@@ -94,6 +94,15 @@ class PullWorkerRuntime:
         self._permanent_error: PullRegistrationError | None = None
         self._rejection_stop_task: asyncio.Task[None] | None = None
         self.max_concurrent_tasks = options.max_concurrent_tasks or os.cpu_count() or 1
+        self.reserved_rpc_tasks = (
+            options.reserved_rpc_tasks
+            if options.reserved_rpc_tasks is not None
+            else int(self.max_concurrent_tasks >= 2)
+        )
+        self._should_warn_rpc_capacity = (
+            options.reserved_rpc_tasks is None and self.max_concurrent_tasks == 1
+        )
+        self.reserved_rpc_declines = 0
         self.idle_streams_per_queue = options.idle_streams_per_queue or 1
         self.stop_drain_timeout = options.stop_drain_timeout.total_seconds() or 30
         self._dispatcher = dispatcher
@@ -132,6 +141,10 @@ class PullWorkerRuntime:
         return None
 
     async def run(self) -> None:
+        if self._should_warn_rpc_capacity:
+            _LOGGER.warning(
+                "RPCs may wait behind long-running Steps; use max_concurrent_tasks >= 2 or set reserved_rpc_tasks explicitly"
+            )
         self._loop = asyncio.get_running_loop()
         self._loop_thread_id = threading.get_ident()
         self._wake = asyncio.Event()
@@ -170,6 +183,8 @@ class PullWorkerRuntime:
     def _reconcile(self, stub: Any) -> None:
         if self._stopping is None or self._stopping.is_set():
             return
+        for stream in tuple(self._streams):
+            stream.try_retire()
         for queue in self.queues:
             idle = sum(
                 stream.queue == queue
@@ -191,11 +206,24 @@ class PullWorkerRuntime:
                 _LOGGER.error("Worker task stream failed", exc_info=failure)
         self.signal()
 
-    def accept_task(self) -> bool:
+    def accept_task(self, task: pb.PollWorkerTask | None = None) -> bool:
         if self._stopping is None or self._stopping.is_set():
             return False
         if self._busy_tasks >= self.max_concurrent_tasks:
             return False
+        if (
+            task is not None
+            and task.WhichOneof("request") in ("wait_for", "execute")
+            and self.reserved_rpc_tasks > 0
+            and self._busy_tasks >= self.max_concurrent_tasks - self.reserved_rpc_tasks
+            and self.unknown_task_reason(task) is None
+        ):
+            self.reserved_rpc_declines += 1
+            request = task.wait_for if task.HasField("wait_for") else task.execute
+            _LOGGER.debug(
+                "decline Step task for reserved RPC capacity: %s", request.flow_type
+            )
+            return False
         self._busy_tasks += 1
         self.signal()
         return True
@@ -279,6 +307,8 @@ class PullStream:
         self.is_busy = False
         self._is_registered = False
         self.is_draining = False
+        self._is_retiring = False
+        self._retirement_requested = asyncio.Event()
         self.is_half_closing = False
         self._active = threading.Event()
         self._call: Any = None
@@ -321,7 +351,11 @@ class PullStream:
                     self.runtime._rejection_stop_task = asyncio.create_task(
                         self.runtime.stop()
                     )
-            if self.runtime.is_stopping or self.runtime._permanent_error is not None:
+            if (
+                self.runtime.is_stopping
+                or self.runtime._permanent_error is not None
+                or self._is_retiring
+            ):
                 break
             if self._is_reconnecting:
                 self._is_reconnecting = False
@@ -378,11 +412,12 @@ class PullStream:
         )
         was_registered = False
         reader: asyncio.Task[Any] | None = None
+        retirement = asyncio.create_task(self._retirement_requested.wait())
         try:
             while not self.runtime.is_stopping or self.is_busy:
                 if reader is None:
                     reader = asyncio.create_task(self._call.read())
-                waiting: set[asyncio.Task[Any]] = {reader}
+                waiting: set[asyncio.Task[Any]] = {reader, retirement}
                 if self.handler_task is not None:
                     waiting.add(self.handler_task)
                 completed, _ = await asyncio.wait(
@@ -390,22 +425,23 @@ class PullStream:
                 )
                 if self.handler_task in completed:
                     await self._finish_handler()
-                    if self.is_draining:
-                        if reader.done():
-                            frame = reader.result()
-                            reader = None
-                            if (
-                                frame is not _GRPC_AIO_EOF
-                                and frame.WhichOneof("frame") == "task"
-                            ):
-                                await self._reject_task(frame.task)
-                        self.is_half_closing = True
-                        self.runtime.signal()
-                        await self._call.done_writing()
-                        await self._await_stream_eof(reader)
+                    self.try_retire()
+                if self.is_draining and not self.is_busy:
+                    if reader.done():
+                        frame = reader.result()
                         reader = None
-                        self._is_reconnecting = True
-                        return was_registered
+                        if (
+                            frame is not _GRPC_AIO_EOF
+                            and frame.WhichOneof("frame") == "task"
+                        ):
+                            await self._reject_task(frame.task)
+                    self.is_half_closing = True
+                    self.runtime.signal()
+                    await self._call.done_writing()
+                    await self._await_stream_eof(reader)
+                    reader = None
+                    self._is_reconnecting = not self._is_retiring
+                    return was_registered
                 if reader not in completed:
                     continue
                 frame = reader.result()
@@ -425,6 +461,7 @@ class PullStream:
                     was_registered = True
                     self._is_registered = True
                     self.runtime.registration.record_registered_stream(self.queue)
+                    self.runtime.signal()
                     continue
                 if kind == "task":
                     await self._start_task(frame.task)
@@ -442,6 +479,8 @@ class PullStream:
                         self._is_reconnecting = True
                         return was_registered
         finally:
+            retirement.cancel()
+            await asyncio.gather(retirement, return_exceptions=True)
             self.is_half_closing = False
             self.cancel_handler()
             if self.handler_task is not None:
@@ -511,7 +550,7 @@ class PullStream:
         if unknown_reason is not None:
             await self._reject_task(task, unknown_reason)
             return
-        if self.is_busy or self.is_draining or not self.runtime.accept_task():
+        if self.is_busy or self.is_draining or not self.runtime.accept_task(task):
             await self._reject_task(task)
             return
         self.is_busy = True
@@ -658,6 +697,22 @@ class PullStream:
         self._active.clear()
         self.runtime.finish_task()
 
+    def try_retire(self) -> None:
+        if self.is_busy or self.is_draining or not self._is_registered:
+            return
+        idle_replacements = sum(
+            candidate is not self
+            and candidate.queue == self.queue
+            and candidate._is_registered
+            and not candidate.is_busy
+            and not candidate.is_draining
+            for candidate in self.runtime._streams
+        )
+        if idle_replacements >= self.runtime.idle_streams_per_queue:
+            self.is_draining = True
+            self._is_retiring = True
+            self._retirement_requested.set()
+
     def cancel_handler(self) -> None:
         self._active.clear()
         if self.runtime._executor is None and self.handler_task is not None:

{dex_python_sdk-1.14.0 → dex_python_sdk-1.15.0}/dex/async_worker.py

@@ -221,6 +221,23 @@ class AsyncWorker:
             raise RuntimeError("wait_pull_registered requires a pull Worker")
         await self._pull.registration.wait_async(timeout)
 
+    @property
+    def pull_reserved_rpc_declines(self) -> int:
+        """Return cumulative Step declines caused by reserved RPC capacity.
+
+        Full-capacity rejections and unknown names are excluded. The count never
+        resets during this Worker's lifetime and stays available after stop.
+        Reading it alongside handlers is safe. Listening Workers return zero.
+        Export this value to a monitoring counter to identify reserve pressure.
+
+        Returns:
+            The cumulative reserve-decline count, or zero for a listening Worker.
+
+        Examples:
+            >>> declines = worker.pull_reserved_rpc_declines
+        """
+        return 0 if self._pull is None else self._pull.reserved_rpc_declines
+
     def pull_registration_state(self) -> PullRegistrationState:
         """Return an immutable snapshot of pull queue registrations.