dex-python-sdk 0.2.8__tar.gz → 0.2.10__tar.gz

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dex-python-sdk
-Version: 0.2.8
+Version: 0.2.10
 Requires-Dist: grpcio>=1.83.0
 Requires-Dist: grpcio-status>=1.83.0
 Requires-Dist: protobuf>=7.35.1
@@ -17,6 +17,47 @@ Project-URL: Repository, https://github.com/superdurable/dex
 
 # Dex SDK for Python
 
+## Pending Channel messages
+
+`Client.get_channel_messages(flow_id, channel)` returns typed `ChannelMessage`
+envelopes in FIFO order. Each envelope contains the decoded value and the UUIDv7
+assigned by Dex. `Client.delete_channel_message` deletes only a still-pending
+message and raises `ChannelMessageNotFoundError` after consumption or another
+deletion.
+
+RPC handlers can stage `channel.delete(context, message_id)`. Declare the RPC as
+`@rpc(is_transactional=True)` when a missing message must abort its other writes.
+Attribute locks already select transactional execution, but Channel deletion
+itself does not.
+
+## RPC state loading
+
+RPCs receive ordinary Attributes and all Channel size metadata by default.
+AttributeMap entries and pending Channel messages are opt-in:
+
+```python
+@dex.rpc(
+    load_attribute_map_instances=(items.load("tenant-a"),),
+    load_channels=(queued,),
+    load_channel_maps=(by_tenant,),
+)
+def snapshot(self, context: dex.Context) -> dex.RPCResult[Snapshot]:
+    messages = queued.pending_messages(context)
+    return dex.RPCResult(Snapshot(messages=messages))
+```
+
+Put an AttributeMap or ChannelMap directly in its plural load option when every
+current instance is needed. Use the singular instance options for the less common
+exact-instance case. A selected empty queue returns an empty tuple; reading an
+unselected map entry or pending-message snapshot raises **StateNotLoadedError**.
+Pending messages preserve FIFO order and include the server-assigned message ID.
+The snapshot does not change after the handler stages a publish or deletion.
+
+Loading controls which data reaches the Worker. **is_transactional** controls
+atomic commit and Channel deletion validation. Attribute locks add isolation
+only among cooperating Steps and RPCs using the same lock. Write-only publish
+and delete operations do not require loading.
+
 Python SDK for [Dex workflow engine](https://github.com/superdurable/dex)
 
 ## New user contracts
@@ -271,14 +312,17 @@ instead of raw Flow, Step, or RPC strings.
 
 ### Waiting and map inspection
 
-`Wait.all_of` and `Wait.any_of` may use unnamed Conditions. Every Condition in
-`Wait.any_combination_of` must have a non-empty user ID; the same Condition
-instance may appear in multiple combinations.
+`Wait.until`, `Wait.all_of`, and `Wait.any_of` use unnamed Conditions by
+default. Do not add condition IDs merely because a Condition is nested in one
+of these waits. Every Condition in `Wait.any_combination_of` must have a
+non-empty user ID; the same Condition instance may appear in multiple
+combinations.
 
 Both `Client` and `AsyncClient` provide singleton and AttributeMap-instance
 overloads of `wait_for_attribute_equal`. They target the current run and accept
 only string, bool, int, or float wire values. JSON objects, bytes, and null fail
-before transport. `AttributeMap.get_map_size/get_all_instance_keys` include
+before transport. Every AttributeMap and ChannelMap instance must be non-empty
+and must not contain `/`. `AttributeMap.get_map_size/get_all_instance_keys` include
 buffered sets and deletes. The matching `ChannelMap` methods are RPC-only,
 include buffered publishes, and omit empty instances. Keys are decoded and
 sorted. Use `force_complete_if_channels_empty(...)` for conditional completion.

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/README.md

@@ -1,6 +1,47 @@
 
 # Dex SDK for Python
 
+## Pending Channel messages
+
+`Client.get_channel_messages(flow_id, channel)` returns typed `ChannelMessage`
+envelopes in FIFO order. Each envelope contains the decoded value and the UUIDv7
+assigned by Dex. `Client.delete_channel_message` deletes only a still-pending
+message and raises `ChannelMessageNotFoundError` after consumption or another
+deletion.
+
+RPC handlers can stage `channel.delete(context, message_id)`. Declare the RPC as
+`@rpc(is_transactional=True)` when a missing message must abort its other writes.
+Attribute locks already select transactional execution, but Channel deletion
+itself does not.
+
+## RPC state loading
+
+RPCs receive ordinary Attributes and all Channel size metadata by default.
+AttributeMap entries and pending Channel messages are opt-in:
+
+```python
+@dex.rpc(
+    load_attribute_map_instances=(items.load("tenant-a"),),
+    load_channels=(queued,),
+    load_channel_maps=(by_tenant,),
+)
+def snapshot(self, context: dex.Context) -> dex.RPCResult[Snapshot]:
+    messages = queued.pending_messages(context)
+    return dex.RPCResult(Snapshot(messages=messages))
+```
+
+Put an AttributeMap or ChannelMap directly in its plural load option when every
+current instance is needed. Use the singular instance options for the less common
+exact-instance case. A selected empty queue returns an empty tuple; reading an
+unselected map entry or pending-message snapshot raises **StateNotLoadedError**.
+Pending messages preserve FIFO order and include the server-assigned message ID.
+The snapshot does not change after the handler stages a publish or deletion.
+
+Loading controls which data reaches the Worker. **is_transactional** controls
+atomic commit and Channel deletion validation. Attribute locks add isolation
+only among cooperating Steps and RPCs using the same lock. Write-only publish
+and delete operations do not require loading.
+
 Python SDK for [Dex workflow engine](https://github.com/superdurable/dex)
 
 ## New user contracts
@@ -255,14 +296,17 @@ instead of raw Flow, Step, or RPC strings.
 
 ### Waiting and map inspection
 
-`Wait.all_of` and `Wait.any_of` may use unnamed Conditions. Every Condition in
-`Wait.any_combination_of` must have a non-empty user ID; the same Condition
-instance may appear in multiple combinations.
+`Wait.until`, `Wait.all_of`, and `Wait.any_of` use unnamed Conditions by
+default. Do not add condition IDs merely because a Condition is nested in one
+of these waits. Every Condition in `Wait.any_combination_of` must have a
+non-empty user ID; the same Condition instance may appear in multiple
+combinations.
 
 Both `Client` and `AsyncClient` provide singleton and AttributeMap-instance
 overloads of `wait_for_attribute_equal`. They target the current run and accept
 only string, bool, int, or float wire values. JSON objects, bytes, and null fail
-before transport. `AttributeMap.get_map_size/get_all_instance_keys` include
+before transport. Every AttributeMap and ChannelMap instance must be non-empty
+and must not contain `/`. `AttributeMap.get_map_size/get_all_instance_keys` include
 buffered sets and deletes. The matching `ChannelMap` methods are RPC-only,
 include buffered publishes, and omit empty instances. Keys are decoded and
 sorted. Use `force_complete_if_channels_empty(...)` for conditional completion.

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/__init__.py

@@ -16,10 +16,11 @@ from dex.attribute import (
     AttributeIndex,
     AttributeLock,
     AttributeMap,
+    AttributeMapLoad,
     IndexType,
 )
 from dex.blob_cache import BlobCache, BlobCacheConfig, open_blob_cache
-from dex.channel import Channel, ChannelMap
+from dex.channel import Channel, ChannelMap, ChannelMapLoad, ChannelMessage
 from dex.client import Client
 from dex.client_options import ClientOptions
 from dex.codec import (
@@ -68,7 +69,9 @@ from dex.runtime_errors import (
     FlowNotFoundError,
     InvalidStepResultError,
     LongPollTimeoutError,
+    ChannelMessageNotFoundError,
     RpcLockConflictError,
+    StateNotLoadedError,
     ValueMappingError,
     WorkerInvocationError,
 )
@@ -115,6 +118,7 @@ __all__ = [
     "AttributeIndex",
     "AttributeLock",
     "AttributeMap",
+    "AttributeMapLoad",
     "AsyncContext",
     "AsyncClient",
     "AsyncBufferedTextStream",
@@ -124,6 +128,9 @@ __all__ = [
     "BufferedTextStream",
     "Channel",
     "ChannelMap",
+    "ChannelMapLoad",
+    "ChannelMessage",
+    "ChannelMessageNotFoundError",
     "Client",
     "ClientOptions",
     "Codec",
@@ -153,6 +160,7 @@ __all__ = [
     "RPCResult",
     "RetryAfterError",
     "RpcLockConflictError",
+    "StateNotLoadedError",
     "Registry",
     "TimeTravelOptions",
     "TimeTravelStepMethod",

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_async_value_hydrator.py

@@ -124,10 +124,24 @@ class AsyncValueHydrator:
         result = pb.InvokeWorkerRPCRequest()
         result.CopyFrom(request)
         values = [request.input, *(entry.value for entry in request.attributes)]
+        channel_messages = [
+            message
+            for channel_values in request.loaded_channel_messages.values()
+            for message in channel_values.messages
+        ]
+        values.extend(message.value for message in channel_messages)
         hydrated = await self.hydrate_all(values)
         result.input.CopyFrom(hydrated[0])
-        for entry, value in zip(result.attributes, hydrated[1:]):
-            entry.value.CopyFrom(value)
+        hydrated_entries = iter(hydrated[1:])
+        for entry in result.attributes:
+            entry.value.CopyFrom(next(hydrated_entries))
+        result_messages = [
+            message
+            for channel_values in result.loaded_channel_messages.values()
+            for message in channel_values.messages
+        ]
+        for message in result_messages:
+            message.value.CopyFrom(next(hydrated_entries))
         return result
 
     async def step_outputs(

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_async_worker_dispatcher.py

@@ -294,6 +294,10 @@ class AsyncWorkerDispatcher(WorkerDispatcher):
             self._values,
             request.attributes,
             channel_infos=dict(request.channel_infos),
+            loaded_channel_messages=dict(request.loaded_channel_messages),
+            loaded_attribute_map_instances=request.loaded_attribute_map_instances,
+            loaded_channel_names=request.loaded_channel_names,
+            loaded_channel_map_instances=request.loaded_channel_map_instances,
             is_active=is_active,
         )
         arguments: list[object] = [context]
@@ -306,6 +310,7 @@ class AsyncWorkerDispatcher(WorkerDispatcher):
             response = pb.InvokeWorkerRPCResponse(
                 upsert_attributes=list(context.attribute_writes.values()),
                 record_events=context.events,
+                delete_from_channel=context.channel_deletions,
                 publish_to_channel=context.publications,
             )
             if isinstance(returned, RPCResult):

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_grpc_errors.py

@@ -19,6 +19,7 @@ from grpc_status import rpc_status
 
 from dex.dexpb import dex_pb2 as pb
 from dex.runtime_errors import (
+    ChannelMessageNotFoundError,
     DexServiceError,
     ErrorSubStatus,
     FlowAlreadyStartedError,
@@ -136,6 +137,8 @@ def translate_rpc_error(
         )
     if sub_status is ErrorSubStatus.LONG_POLL_TIMEOUT:
         return LongPollTimeoutError(*parameters)
+    if sub_status is ErrorSubStatus.CHANNEL_MESSAGE_NOT_FOUND:
+        return ChannelMessageNotFoundError(*parameters)
     return DexServiceError(*parameters)
 
 
@@ -219,5 +222,8 @@ def _map_sub_status(value: int) -> ErrorSubStatus:
         int(pb.ERROR_SUB_STATUS_FLOW_NOT_EXISTS): ErrorSubStatus.FLOW_NOT_EXISTS,
         int(pb.ERROR_SUB_STATUS_WORKER_API_ERROR): ErrorSubStatus.WORKER_API_ERROR,
         int(pb.ERROR_SUB_STATUS_LONG_POLL_TIME_OUT): ErrorSubStatus.LONG_POLL_TIMEOUT,
+        int(
+            pb.ERROR_SUB_STATUS_CHANNEL_MESSAGE_NOT_FOUND
+        ): ErrorSubStatus.CHANNEL_MESSAGE_NOT_FOUND,
     }
     return statuses.get(value, ErrorSubStatus.UNCATEGORIZED)

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_invocation_context.py

@@ -12,14 +12,15 @@ from enum import Enum
 from typing import Any, Callable, Protocol, Sequence, TypeVar, cast
 from urllib.parse import unquote
 
-from dex._utils import require_name
+from dex._utils import require_map_instance, require_name
 from dex._value_mapper import ValueMapper
 from dex.attribute import Attribute, AttributeMap, _apply_attribute_store_sync
-from dex.channel import Channel, ChannelMap
+from dex.channel import Channel, ChannelMap, ChannelMessage
 from dex.codec import Codec
 from dex.dexpb import dex_pb2 as pb
 from dex.flow import Registry, _RegisteredFlow
 from dex.flow_result import FlowResult, flow_result_from_proto
+from dex.runtime_errors import StateNotLoadedError
 from dex.step import (
     _NO_HEARTBEAT_VALUE,
     StepOutput,
@@ -66,6 +67,10 @@ class InvocationContext:
         locals: Sequence[pb.KV] = (),
         condition_results: pb.ConditionResults | None = None,
         channel_infos: dict[str, pb.ChannelInfo] | None = None,
+        loaded_channel_messages: dict[str, pb.ChannelValues] | None = None,
+        loaded_attribute_map_instances: Sequence[str] = (),
+        loaded_channel_names: Sequence[str] = (),
+        loaded_channel_map_instances: Sequence[str] = (),
         is_active: Callable[[], bool] | None = None,
         output_emitter: _StepOutputEmitter | None = None,
     ) -> None:
@@ -77,11 +82,16 @@ class InvocationContext:
         self._locals = self._map_values("step-execution local", locals)
         self._condition_results = condition_results
         self._channel_infos = dict(channel_infos or {})
+        self._loaded_channel_messages = dict(loaded_channel_messages or {})
+        self._loaded_attribute_map_instances = frozenset(loaded_attribute_map_instances)
+        self._loaded_channel_names = frozenset(loaded_channel_names)
+        self._loaded_channel_map_instances = frozenset(loaded_channel_map_instances)
         self._is_active = is_active or _always_active
         self._output_emitter = output_emitter
         self.attribute_writes: dict[str, pb.AttributeWrite] = {}
         self.local_writes: dict[str, pb.KV] = {}
         self.events: list[pb.KV] = []
+        self.channel_deletions: list[pb.ChannelMessageDeletion] = []
         self.publications: list[pb.ChannelMessage] = []
         self._event_names: set[str] = set()
         self._step_output_finalizers: list[_StepOutputFinalizer] = []
@@ -275,6 +285,7 @@ class InvocationContext:
         instance: str | None,
     ) -> ValueT:
         self._require_registered(definition)
+        self._require_attribute_map_instance_loaded(definition, instance)
         key = self._physical_name(definition, instance)
         write = self.attribute_writes.get(key)
         if write is not None:
@@ -349,11 +360,32 @@ class InvocationContext:
                 size=(current.size if current is not None else 0) + 1
             )
 
+    def _delete_channel_message(
+        self,
+        definition: Channel[object] | ChannelMap[object],
+        instance: str | None,
+        message_id: str,
+    ) -> None:
+        if self._method is not InvocationMethod.RPC:
+            raise ValueError("Channel message deletion requires an RPC Context")
+        self._require_registered(definition)
+        name = self._physical_name(definition, instance)
+        self.channel_deletions.append(
+            pb.ChannelMessageDeletion(
+                channel_name=name,
+                message_id=require_name(message_id),
+            )
+        )
+        current = self._channel_infos.get(name)
+        if current is not None and current.size > 0:
+            self._channel_infos[name] = pb.ChannelInfo(size=current.size - 1)
+
     def _attribute_map_keys(
         self,
         definition: AttributeMap[object],
     ) -> tuple[str, ...]:
         self._require_registered(definition)
+        self._require_attribute_map_all_loaded(definition)
         prefix = f"{definition.name}/"
         physical_keys = {key for key in self._attributes if key.startswith(prefix)}
         for key, write in self.attribute_writes.items():
@@ -363,7 +395,12 @@ class InvocationContext:
                 physical_keys.discard(key)
             else:
                 physical_keys.add(key)
-        return tuple(sorted(unquote(key[len(prefix) :]) for key in physical_keys))
+        return tuple(
+            sorted(
+                require_map_instance(unquote(key[len(prefix) :]))
+                for key in physical_keys
+            )
+        )
 
     def _channel_map_keys(
         self,
@@ -375,7 +412,7 @@ class InvocationContext:
         prefix = f"{definition.name}/"
         return tuple(
             sorted(
-                unquote(key[len(prefix) :])
+                require_map_instance(unquote(key[len(prefix) :]))
                 for key, info in self._channel_infos.items()
                 if key.startswith(prefix) and info.size > 0
             )
@@ -390,6 +427,39 @@ class InvocationContext:
         info = self._channel_infos.get(self._physical_name(definition, instance))
         return info.size if info is not None else 0
 
+    def _pending_channel_messages(
+        self,
+        definition: Channel[ValueT] | ChannelMap[ValueT],
+        instance: str | None,
+    ) -> tuple[ChannelMessage[ValueT], ...]:
+        if self._method is not InvocationMethod.RPC:
+            raise ValueError("pending Channel messages require an RPC Context")
+        self._require_registered(definition)
+        if isinstance(definition, ChannelMap):
+            channel_name = self._physical_name(definition, instance)
+            is_loaded = (
+                f"{definition.name}/" in self._loaded_channel_map_instances
+                or channel_name in self._loaded_channel_map_instances
+            )
+        else:
+            channel_name = self._physical_name(definition, instance)
+            is_loaded = definition.name in self._loaded_channel_names
+        if not is_loaded:
+            raise StateNotLoadedError(
+                f"Channel messages were not loaded for RPC: {definition.name}"
+            )
+        values = self._loaded_channel_messages.get(channel_name)
+        if values is None:
+            return ()
+        codec = self._flow_codec(definition.value_type)
+        return tuple(
+            ChannelMessage(
+                message_id=message.message_id,
+                value=cast(ValueT, self._values.decode(message.value, codec)),
+            )
+            for message in values.messages
+        )
+
     def _channel_results(
         self,
         definition: Channel[ValueT] | ChannelMap[ValueT],
@@ -411,6 +481,36 @@ class InvocationContext:
     def _flow_codec(self, value_type: object) -> Codec[Any]:
         return self._values.codec(value_type)
 
+    def _require_attribute_map_instance_loaded(
+        self,
+        definition: Attribute[Any] | AttributeMap[Any],
+        instance: str | None,
+    ) -> None:
+        if self._method is not InvocationMethod.RPC or not isinstance(
+            definition, AttributeMap
+        ):
+            return
+        physical_name = self._physical_name(definition, instance)
+        if (
+            f"{definition.name}/" not in self._loaded_attribute_map_instances
+            and physical_name not in self._loaded_attribute_map_instances
+        ):
+            raise StateNotLoadedError(
+                f"AttributeMap instance was not loaded for RPC: {physical_name}"
+            )
+
+    def _require_attribute_map_all_loaded(
+        self,
+        definition: AttributeMap[object],
+    ) -> None:
+        if (
+            self._method is InvocationMethod.RPC
+            and f"{definition.name}/" not in self._loaded_attribute_map_instances
+        ):
+            raise StateNotLoadedError(
+                f"all AttributeMap instances were not loaded for RPC: {definition.name}"
+            )
+
     def _require_registered(self, definition: _Definition) -> None:
         registered = self._flow.persistence.get(definition.name)
         if registered is not definition:

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_utils.py

@@ -13,6 +13,20 @@ def require_name(name: str) -> str:
     return name
 
 
+def require_persistence_definition_name(name: str) -> str:
+    require_name(name)
+    if "/" in name:
+        raise ValueError("persistence definition names must not contain '/'")
+    return name
+
+
+def require_map_instance(instance: str) -> str:
+    require_name(instance)
+    if "/" in instance:
+        raise ValueError("map instances must not contain '/'")
+    return instance
+
+
 def validate_condition_id(condition_id: str | None) -> None:
     if condition_id is not None and not condition_id:
         raise ValueError("condition ID must not be empty")

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_value_hydrator.py

@@ -131,10 +131,24 @@ class ValueHydrator:
         result = pb.InvokeWorkerRPCRequest()
         result.CopyFrom(request)
         values = [request.input, *(entry.value for entry in request.attributes)]
+        channel_messages = [
+            message
+            for channel_values in request.loaded_channel_messages.values()
+            for message in channel_values.messages
+        ]
+        values.extend(message.value for message in channel_messages)
         hydrated = self.hydrate_all(values)
         result.input.CopyFrom(hydrated[0])
-        for entry, value in zip(result.attributes, hydrated[1:]):
-            entry.value.CopyFrom(value)
+        hydrated_entries = iter(hydrated[1:])
+        for entry in result.attributes:
+            entry.value.CopyFrom(next(hydrated_entries))
+        result_messages = [
+            message
+            for channel_values in result.loaded_channel_messages.values()
+            for message in channel_values.messages
+        ]
+        for message in result_messages:
+            message.value.CopyFrom(next(hydrated_entries))
         return result
 
     def step_outputs(

{dex_python_sdk-0.2.8 → dex_python_sdk-0.2.10}/dex/_worker_dispatcher.py

@@ -219,6 +219,10 @@ class WorkerDispatcher:
             self._values,
             request.attributes,
             channel_infos=dict(request.channel_infos),
+            loaded_channel_messages=dict(request.loaded_channel_messages),
+            loaded_attribute_map_instances=request.loaded_attribute_map_instances,
+            loaded_channel_names=request.loaded_channel_names,
+            loaded_channel_map_instances=request.loaded_channel_map_instances,
             is_active=is_active,
         )
         arguments: list[object] = [context]
@@ -233,6 +237,7 @@ class WorkerDispatcher:
             response = pb.InvokeWorkerRPCResponse(
                 upsert_attributes=list(context.attribute_writes.values()),
                 record_events=context.events,
+                delete_from_channel=context.channel_deletions,
                 publish_to_channel=context.publications,
             )
             if isinstance(returned, RPCResult):