loom-kernel 6.0.0__tar.gz → 6.0.2__tar.gz

{loom_kernel-6.0.0 → loom_kernel-6.0.2}/CHANGELOG.md

@@ -920,6 +920,88 @@
   `BROADCAST_NOT_LAST` with component `broadcast`, at the top level, in a
   branch and in a Router branch. It used to report `TERMINAL_NOT_LAST` with
   component `Broadcast`.
+- **streaming:** a Router branch that ends in `IntoTopic`, `IntoTable` (any
+  `IntoSink`) or `Drain` hands the message to that terminal. The branch's
+  terminal was skipped: with a flow `output` the message went to the output
+  instead, without one it was not written.
+- **streaming:** `With`, `WithAsync`, `Router`, `CollectBatch`, `ForEach` and
+  `Explode` inside a Router branch fail compilation with
+  `ROUTER_BRANCH_UNSAFE_NODE`. Such flows compiled and raised `TypeError` when
+  the branch ran.
+- **streaming:** a `Router` requires a record stream. A batch reaching a
+  `Router`, such as after a `CollectBatch`, fails compilation with
+  `SHAPE_MISMATCH` before `Router`. Such flows compiled and raised
+  `TypeError` at the Router when a batch arrived.
+- **streaming:** `ROUTER_BRANCH_SHAPE_DIVERGENCE` reads "messages continuing
+  after the router have different shapes: ...", since the unmatched input of
+  a `Router` without a default takes part in the check. It read "router
+  branches produce different shapes: ...".
+- **streaming:** `SinkPartition.write_batch` and `close` return an `int`: how
+  many of the oldest events not yet reported durable they made durable, `0`
+  when they only buffered. A custom `IntoSink` partition must report it: a
+  `write_batch` that returns anything else, a `bool` included, fails with
+  `TypeError`, and a bad count from `close()` is logged as
+  `storage_sink_close_count_invalid` and completes nothing. After a call
+  raised or returned a refused count, later counts complete nothing. A storage sink's
+  `terminal:sink_write` spans are emitted for the events a call made durable,
+  not for every event handed to it. A buffering partition migrates like this:
+
+  ```python
+  # before
+  def write_batch(self, items):
+      self._buffer.extend(items)
+
+  def close(self):
+      flush(self._path, self._buffer)
+
+  # after
+  def write_batch(self, items) -> int:
+      self._buffer.extend(items)
+      return 0  # buffered, nothing durable yet
+
+  def close(self) -> int:
+      flush(self._path, self._buffer)
+      flushed, self._buffer = len(self._buffer), []
+      return flushed  # durable now
+  ```
+- **streaming:** under at-least-once delivery, a caller-supplied terminal sink
+  written inline by `With(process=...)` or `WithAsync(process=...)` must expose
+  `build_inline_partition()`, which returns a partition that completes no
+  offset and records no death; without it the flow fails at assembly with
+  `RuntimeConfigurationError` (`INLINE_SINK_CANNOT_SKIP_COMMITS`). Its
+  partitions completed offsets at the inline write.
+- **streaming:** a Delta `IntoTable` whose payload type has no fields fails
+  with `ValueError` when its partition is built. Its items wrote no row.
+- **streaming:** an output declared after a node that ends every path fails
+  compilation. A flow `output` after a process that ends in `IntoTopic`,
+  `IntoSink`, `Drain`, `ExpandRoutes`, or a `Router` with a `default` whose
+  branches all end in one of `IntoTopic`, `IntoSink`, `Drain`, fails with
+  `OUTPUT_AFTER_PATH_END`. A Broadcast route `output` after a process that ends
+  in any of those, in a `Broadcast`, or in a `Fork` with a `default` fails
+  with `BROADCAST_OUTPUT_AFTER_PATH_END`. Such flows compiled and wrote the
+  message twice, handed the expanded payload of an `ExpandRoutes` to the
+  output, or never wrote the output. A step or an `IntoTopic` after such a
+  `Router` fails with `TERMINAL_NOT_LAST`, the code an `IntoTopic`, `IntoSink`
+  or `Drain` followed by a node reports. A
+  branch that ends without a sink no longer logs `unrouted_branch_drop_sink`.
+- **streaming:** a `terminal_sinks` override passed to `build_dataflow` must
+  cover every compiled `IntoTopic` path; a missing one fails with
+  `RuntimeConfigurationError` (`TERMINAL_SINK_MISSING`). The path fell back to
+  a drop sink, so its messages completed without being written.
+- **streaming:** a failed send or flush to a Kafka sink's DLQ topic fails the
+  run (inside `With` or `WithAsync`, the message becomes an error envelope). It
+  used to be logged as `dlq_send_failed` and the batch's offsets were
+  committed. The fix is under Fixes.
+- **streaming:** a DLQ record is built from the same send request as the sink
+  record: it keeps the message key (and with it the partition and order), the
+  message type and the headers of the sink record, error envelopes now carry
+  `x-error-reason` and follow the sink's partition policy, and `x-dlq-error`
+  is added. A diverted message keeps its own trace id and parent trace id; an
+  error envelope without an original message gets a new trace id, as on the
+  sink, instead of none.
+- **streaming:** a sync `With(process=...)` hands on one message at a time, as
+  `WithAsync` does. After a `CollectBatch` it handed on whole batches, so as
+  the last node of a flow without `output` its messages never completed.
 
 ## 🐛 Fixes
 
@@ -1242,6 +1324,49 @@
   follows. Each route is now checked on the record stream it receives, with
   issues scoped under `expand_routes branch <type>`, and a node after an
   `ExpandRoutes` fails with `TERMINAL_NOT_LAST`; such flows used to compile.
+- **streaming:** under at-least-once delivery each source offset completes
+  once per fork share, and not before the writes on its path are flushed.
+  These shapes broke it:
+  - an inline `IntoTopic` in `With(process=...)` or `WithAsync(process=...)`
+    followed by a flow `output` completed the offset at the inline write,
+    before the output write, and again at the output;
+  - an `ExpandRoutes` route ending in a scope completed at a drop sink wired
+    beside the route's process, which does not wait for the inline write, and
+    again at the inline write;
+  - an `IntoTopic` ending an `ExpandRoutes` route, or a Broadcast route
+    without `output`, completed at its sink and again at a drop sink;
+  - a Router branch terminal, and a Router's unmatched messages with nothing
+    after them, never completed;
+  - a storage sink (`IntoTable`, any `IntoSink`) never completed;
+  - an undecodable record with no `WIRE` error route never completed;
+  - a batch diverted to a DLQ topic completed before the DLQ records were
+    flushed.
+
+  A Router keeps the source order of the messages that continue after it, and
+  an inline write no longer completes an offset: the message completes where
+  its path ends. A storage sink completes the offsets of the events its flush
+  wrote, so events still buffered when a worker dies are consumed again.
+- **streaming:** a sync `With(process=...)` runs on a record stream. It
+  accepted only batches and raised `TypeError` for every message.
+- **streaming:** a Kafka sink batch diverted to its DLQ topic commits its
+  offsets only once the DLQ topic received every record. The diversion sent
+  each record without flushing, logged and swallowed per-record failures, and
+  committed anyway, so a batch the DLQ topic never received was lost. The
+  producer is now flushed after the diversion, and a failed DLQ send or flush
+  raises `KafkaDeliveryError` naming the DLQ topic, with the failure the DLQ
+  step observed as its cause. The batch is not committed and is handled like a failed batch on a
+  sink without a DLQ topic: the run fails and, on restart, the batch is
+  consumed again from the recovery snapshot or, without one, from the last
+  committed offset; inside `With` or `WithAsync` the message becomes an error
+  envelope.
+- **streaming:** a `Router` branch that ends its path in `IntoTopic`,
+  `IntoSink` or `Drain` no longer takes part in the branch shape check, and a
+  `Router` reports the shape of the messages that continue after it: the
+  output of the branches that do not end their path and, without a default
+  branch, the unmatched input. A `Router` with an `IntoTopic` branch and a
+  `Drain` default failed with `ROUTER_BRANCH_SHAPE_DIVERGENCE`, and a step
+  after a `Router` without a default whose branches all end their path failed
+  with `SHAPE_MISMATCH`; both compile now.
 
 ### prefect
 

{loom_kernel-6.0.0 → loom_kernel-6.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 6.0.0
+Version: 6.0.2
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3

{loom_kernel-6.0.0 → loom_kernel-6.0.2}/src/loom/streaming/bytewax/_adapter.py

@@ -3,6 +3,20 @@
 Translates a :class:`CompiledPlan` into a Bytewax :class:`Dataflow`,
 wiring decode, node dispatch, encode, and output routing operators.
 
+Offset completion invariant: for every flow that compiles, each source offset,
+or each share of it a fan-out node forked, completes exactly once, at the end
+of the path it travels and after that path's last durable write. A node
+handler returns the stream that is still live after it, or ``None`` when it
+closed every path it received. ``IntoTopic`` ends its
+path at its compiled sink through :meth:`_BuildContext.wire_branch_terminal`.
+A branching node wires each branch through :meth:`_BuildContext.wire_branch`,
+which hands the branch's live end to ``wire_branch_terminal``: the sink
+compiled for the branch, else a drop sink. The flow's own live end goes
+through :meth:`_BuildContext.wire_flow_output`: the flow output, else a drop
+sink. A Bytewax output has no downstream, so no completion relies on the order
+of sibling operators: it happens inside the sink that writes, or downstream of
+an inline write.
+
 Requires ``bytewax`` to be installed.
 """
 
@@ -63,11 +77,13 @@ class _BuiltDataflow:
 
 
 class _DropSinkPartition(StatelessSinkPartition[Any]):
-    """Discard items routed to an unrouted error branch.
+    """Discard the items of a path that ends without a sink.
 
-    Discarding still completes each record with the commit tracker: a dropped
-    message that never completes would freeze its partition's watermark under
-    at-least-once delivery.
+    Such a path is an error kind with no route, a branch or flow end that owns
+    no output, or an undecodable record with no WIRE route. Discarding still
+    completes each record with the commit tracker: a dropped message that never
+    completes would freeze its partition's watermark under at-least-once
+    delivery.
     """
 
     def __init__(self, commit_tracker: CommitCompletionPort | None = None) -> None:
@@ -83,7 +99,7 @@ class _DropSinkPartition(StatelessSinkPartition[Any]):
 
 
 class _DropSink(DynamicSink[Any]):
-    """Build a tracker-aware discard sink for unrouted error branches."""
+    """Build the tracker-aware discard sink for a path that ends without a sink."""
 
     def __init__(self, commit_tracker: CommitCompletionPort | None = None) -> None:
         self._commit_tracker = commit_tracker
@@ -182,18 +198,14 @@ class _SupportsTerminalTracing(Protocol):
     ) -> None:
         """Make this sink emit a terminal span per message it writes."""
 
-    def mark_inline_partition(self) -> None:
-        """Record that an inline, non-terminal partition is built from this sink."""
 
+@runtime_checkable
+class _SupportsInlinePartition(Protocol):
+    """Runtime sink that can build a partition for an inline, non-terminal write."""
 
-def _mark_inline_partition(sink: object) -> None:
-    """Record on one sink that an inline, non-terminal partition is built from it.
-
-    Sinks that cannot carry terminal tracing — storage sinks, test doubles —
-    have nothing to record and are left untouched.
-    """
-    if isinstance(sink, _SupportsTerminalTracing):
-        sink.mark_inline_partition()
+    def build_inline_partition(self) -> StatelessSinkPartition[Any]:
+        """Build a partition that completes no offset and records no death."""
+        ...
 
 
 def _bind_commit_tracker_object(
@@ -235,8 +247,10 @@ def _assemble_dataflow(plan: CompiledPlan, ctx: _BuildContext) -> Any:
     flow = _bytewax_dataflow.Dataflow(plan.name)
     stream = _build_source_pipeline(flow, ctx)
     stream = _wire_process(stream, tuple(node.node for node in plan.nodes), ctx)
-
-    _wire_output(stream, ctx)
+    ctx.wire_flow_output(stream)
+    for kind in plan.error_routes:
+        if kind not in ctx.error_sinks:
+            raise RuntimeError(f"Bytewax sink is required for error route {kind.value}.")
     return flow
 
 
@@ -281,43 +295,66 @@ class _BuildContext:
         self.resource_manager = ResourceManager(bridge)
         self._path: tuple[int, ...] = ()
 
-    def wire_terminal(self, step_id: str, stream: Any) -> None:
-        if self.sink is None:
-            raise RuntimeError("Bytewax sink is required for terminal output wiring.")
-        self._bind_terminal_tracing(self.sink)
-        bw_output(step_id, stream, self.sink)
+    def wire_branch(
+        self,
+        step_id: str,
+        stream: Any,
+        nodes: tuple[object, ...],
+        path: tuple[int, ...],
+        *,
+        start: int = 0,
+    ) -> None:
+        """Wire one branch's *nodes* from index *start* under *path*; end what stays live."""
+        tail = _wire_process(stream, nodes, self, path_prefix=path, start=start)
+        self.wire_branch_terminal(step_id, tail, path)
 
-    def wire_branch_terminal(self, step_id: str, stream: Any, path: tuple[int, ...]) -> None:
-        sink = self.terminal_sinks.get(path)
+    def wire_branch_terminal(self, step_id: str, stream: Any | None, path: tuple[int, ...]) -> None:
+        """End one branch's live stream at the sink compiled for *path*, else a drop sink.
+
+        ``None`` means the branch already ended in a sink, which completes what
+        it writes; wiring anything more would complete those offsets twice.
+        """
+        sink = self._runtime_sink_for(path)
+        if stream is None:
+            if sink is not None:
+                raise RuntimeError(
+                    f"the output compiled for path {path} is never written: the branch "
+                    "already ended the path of every message it received."
+                )
+            return
         if sink is not None:
             self._bind_terminal_tracing(sink)
             bw_output(_qualified_step_id(step_id, path), stream, sink)
             return
-        # A branch without a terminal still owes its record a completion: the
-        # broadcast that created it already forked the offset once per branch.
-        # Discarding the stream silently left that fork outstanding forever and
-        # froze the partition's watermark. Unrouted error branches already fall
-        # back to a drop sink for exactly this reason; branches now do too.
-        logger.warning(
-            "unrouted_branch_drop_sink",
-            flow=self.plan.name,
-            step_id=step_id,
-            path=path,
-        )
+        # A live branch end with no sink still owes each record a completion;
+        # discarding it would leave the offset, or its fork share, outstanding
+        # and freeze the partition's watermark.
         bw_output(
             _qualified_step_id(f"{step_id}_dropped", path),
             stream,
             _DropSink(self.commit_tracker),
         )
 
+    def _runtime_sink_for(self, path: tuple[int, ...]) -> Any | None:
+        """Return the runtime sink for *path*, or ``None`` when nothing was compiled there.
+
+        Raises:
+            RuntimeConfigurationError: If an ``IntoTopic`` was compiled at *path*
+                but no runtime sink was supplied for it.
+        """
+        sink = self.terminal_sinks.get(path)
+        compiled = self.plan.terminal_sinks.get(path)
+        if sink is None and compiled is not None:
+            issue = StreamingErrorCode.TERMINAL_SINK_MISSING.issue(path=path, topic=compiled.topic)
+            raise RuntimeConfigurationError([issue])
+        return sink
+
     def _bind_terminal_tracing(self, sink: Any) -> None:
         """Give one real terminal sink the runtime its death spans open on.
 
         Called only from the two wiring sites that run for a genuine terminal.
-        ``_BuildContext.inline_sink_partition_for`` builds from the *same*
-        ``terminal_sinks`` mapping for ``WithAsync(process=[..., IntoTopic])``,
-        where the message explicitly does not die; binding any earlier would
-        emit a death span for a message that goes on living.
+        An inline partition comes from ``build_inline_partition``, which never
+        applies tracing, because the message goes on after an inline write.
 
         Sinks that cannot record a death — storage sinks, drop sinks, test
         doubles — are left untouched.
@@ -338,22 +375,26 @@ class _BuildContext:
         )
         bw_output(f"{step_id}_{kind.value}_dropped", stream, _DropSink(self.commit_tracker))
 
-    def wire_flow_output(self, stream: Any, plan: CompiledPlan) -> None:
-        if self.sink is None and plan.output is not None:
+    def wire_flow_output(self, stream: Any | None) -> None:
+        """End the flow's live stream at the output sink, else complete it at a drop sink."""
+        if self.sink is None and self.plan.output is not None:
             raise RuntimeError("Bytewax sink is required for terminal output wiring.")
-        if self.sink is not None:
-            self.wire_terminal("output", stream)
-        for kind in plan.error_routes:
-            if kind not in self.error_sinks:
-                raise RuntimeError(f"Bytewax sink is required for error route {kind.value}.")
-
-    def wire_decode_error(self, stream: Any, plan: CompiledPlan) -> None:
-        del plan
-        if self.error_sinks.get(ErrorKind.WIRE) is not None:
-            self.wire_node_error(ErrorKind.WIRE, "decode", stream)
-            return
-        if ErrorKind.WIRE in self.plan.error_routes:
+        if stream is None and self.plan.output is not None:
+            raise RuntimeError(
+                "flow.output is never written: the process already ended the path of every "
+                "message it received."
+            )
+        if stream is not None and self.sink is not None:
+            self._bind_terminal_tracing(self.sink)
+            bw_output("output", stream, self.sink)
+        elif stream is not None:
+            bw_output("output_dropped", stream, _DropSink(self.commit_tracker))
+
+    def wire_decode_error(self, stream: Any) -> None:
+        """Route undecodable records to the WIRE error sink, else complete them at a drop sink."""
+        if ErrorKind.WIRE in self.plan.error_routes and ErrorKind.WIRE not in self.error_sinks:
             raise RuntimeError("Bytewax sink is required for WIRE error routing.")
+        self.wire_node_error(ErrorKind.WIRE, "decode", stream)
 
     def inline_sink_partition_for(
         self,
@@ -373,16 +414,27 @@ class _BuildContext:
             sink is registered for *path*.
 
         Raises:
-            RuntimeError: If the sink for *path* already carries terminal
-                tracing, meaning an inline and a terminal wiring path collided
-                on one compiled sink.
+            RuntimeConfigurationError: If an ``IntoTopic`` was compiled at *path*
+                but no runtime sink was supplied for it
+                (``TERMINAL_SINK_MISSING``), or if a commit tracker is bound and
+                the sink cannot build an inline partition, so its partitions
+                would complete offsets at the inline write
+                (``INLINE_SINK_CANNOT_SKIP_COMMITS``).
         """
-        sink = self.terminal_sinks.get(path)
+        sink = self._runtime_sink_for(path)
         if sink is None:
             return None
-        _mark_inline_partition(sink)
-        step_id = "inline_" + "_".join(str(p) for p in path)
-        return cast(StatelessSinkPartition[Any], sink.build(step_id, 0, 1))
+        if isinstance(sink, _SupportsInlinePartition):
+            return sink.build_inline_partition()
+        if self.commit_tracker is not None:
+            issue = StreamingErrorCode.INLINE_SINK_CANNOT_SKIP_COMMITS.issue(
+                component=type(sink).__name__
+            )
+            raise RuntimeConfigurationError([issue])
+        partition: StatelessSinkPartition[Any] = sink.build(
+            "inline_" + "_".join(str(p) for p in path), 0, 1
+        )
+        return partition
 
     def manager_for(
         self,
@@ -411,16 +463,6 @@ class _BuildContext:
         finally:
             self._path = previous
 
-    def wire_process(
-        self,
-        stream: Any,
-        nodes: tuple[object, ...],
-        *,
-        path_prefix: tuple[int, ...] = (),
-    ) -> Any:
-        """Wire one nested process subtree."""
-        return _wire_process(stream, nodes, self, path_prefix=path_prefix)
-
     def shutdown_all(self) -> None:
         """Shutdown all resource managers."""
         self.resource_manager.shutdown_all()
@@ -442,7 +484,7 @@ def _build_source_pipeline(flow: Any, ctx: _BuildContext) -> Stream:
     stream: Stream = bw_input("source", flow, source)
     decoded = _decode_source_stream(stream, ctx, codec, step_id)
     decoded_branch = _split_decode_results(decoded, step_id)
-    ctx.wire_decode_error(decoded_branch.falses, ctx.plan)
+    ctx.wire_decode_error(decoded_branch.falses)
     return bw_map(f"{step_id}_message", decoded_branch.trues, _decode_ok_message)
 
 
@@ -461,11 +503,6 @@ def _split_decode_results(stream: Stream, step_id: str) -> Any:
     return branch(f"{step_id}_is_ok", stream, _is_decode_ok)
 
 
-def _wire_output(stream: Any, ctx: _BuildContext) -> None:
-    """Wire the output sink and error routes."""
-    ctx.wire_flow_output(stream, ctx.plan)
-
-
 def _qualified_step_id(step_id: str, path: tuple[int, ...]) -> str:
     if not path:
         return step_id

{loom_kernel-6.0.0 → loom_kernel-6.0.2}/src/loom/streaming/bytewax/_error_boundary.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 from collections.abc import Callable, Sequence
 from dataclasses import dataclass
-from typing import Any, Protocol, TypeAlias
+from typing import Any, Protocol, TypeAlias, TypeVar
 
 from bytewax.operators import branch, flat_map
 
@@ -19,6 +19,7 @@ from loom.streaming.core._typing import StreamPayload
 
 Stream: TypeAlias = Any
 NodeResult: TypeAlias = Message[StreamPayload] | ErrorEnvelope[StreamPayload]
+ResultT = TypeVar("ResultT")
 logger = get_logger(__name__)
 
 
@@ -87,9 +88,9 @@ def _build_error_envelope(
 def _execute_in_boundary(
     classify: Callable[[Exception], ErrorKind],
     original: Message[StreamPayload],
-    fn: Callable[[], Message[StreamPayload]],
+    fn: Callable[[], ResultT],
     boundary: ErrorBoundary,
-) -> NodeResult:
+) -> ResultT | ErrorEnvelope[StreamPayload]:
     """Execute one message node and capture failures as error envelopes."""
     try:
         return fn()