loom-kernel 7.0.0__tar.gz → 8.0.1__tar.gz

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/CHANGELOG.md

@@ -459,6 +459,16 @@
 
 ### core
 
+- **core:** a `Decimal` field, plain or optional, declares its column type: a
+  SQLAlchemy model with an undeclared one is refused when it is compiled,
+  naming the field. loom inferred `NUMERIC(10, 2)`, and Postgres silently
+  rounds a value to that scale (`1.2345` is stored as `1.23`) and refuses one
+  past that precision. Migration: declare it, e.g.
+  `Annotated[Decimal, Numeric(precision=20, scale=6)]` or
+  `ColumnField(Numeric(precision=20, scale=6))`; `Numeric()` with no
+  precision is now an unbounded `NUMERIC` instead of `NUMERIC(10, 2)`, and
+  `Numeric(scale=...)` without a precision raises `ValueError`, as SQLAlchemy
+  would drop the scale.
 - **core:** `sql_endpoint.auth` admits only `identity` and `external`;
   `auth: jwt` is refused when the config is loaded, and the alias and its
   `DeprecationWarning` are gone. Migration: replace `auth: jwt` with
@@ -949,6 +959,40 @@
 
 ### streaming
 
+- **streaming:** `FromTopic`, `FromMultiTypeTopic`, `FromMongoCDC` and
+  `IntoTopic` take no `shape`, `CompiledSingleSource`, `CompiledMultiSource`
+  and `CompiledMongoCDCSource` have no `shape` field, and `StreamShape.MANY`
+  is removed. The runtime never read a declared shape: every source emits one
+  record per message, so a source declared `BATCH` was shape-checked as
+  batches while its nodes received records, and an `IntoTopic` writes one
+  record per message whatever it declared. Every flow now starts as a record
+  stream, and an `IntoTopic` ends its path (shape `none`). Migration: drop
+  `shape=`; to work on batches, use `CollectBatch` and, back to records,
+  `ForEach`.
+- **streaming:** a batch reaching a `Drain` or a `Broadcast` fails compilation
+  with `SHAPE_MISMATCH`, whatever the delivery. Under at-most-once delivery
+  `CollectBatch → Drain` compiled and dropped the batch; under at-least-once
+  the batch's offsets never completed. Under at-least-once a `Broadcast` with
+  more than one route raised `TypeError` (`Expected Message, got list`) on a
+  batch. Put a `ForEach` before the node.
+- **streaming:** `BatchExpandStep.execute` returns one list of payloads per
+  message of its batch, in batch order, instead of one flat list. List `i`
+  holds what `messages[i]` expands into; each output takes that message's
+  metadata, and an empty list filters the message out. A result whose length
+  differs from the batch, a flat list of payloads, a `Message` among the
+  outputs, or a `str`, `bytes`, `bytearray` or mapping in place of the result
+  or of one of its lists raises `TypeError`, and the batch becomes error
+  envelopes.
+
+  ```python
+  # before
+  def execute(self, messages: list[Message[Event]], **kwargs: object) -> list[Score]:
+      return [Score(value=m.payload.amount) for m in messages]
+
+  # after
+  def execute(self, messages: list[Message[Event]], **kwargs: object) -> list[list[Score]]:
+      return [[Score(value=m.payload.amount)] for m in messages]
+  ```
 - **streaming:** compile issues come from the code catalogue
   (`StreamingErrorCode.X.issue(...)`), and the compiler's private issue
   factories are removed. `loom.streaming.compiler.CompilationIssue` is a type
@@ -957,9 +1001,11 @@
   `CompileIssue` instead), `loom_message_type()` is gone, and
   `CompilationIssue.prefixed()` is removed. Messages keep their text.
 - **streaming:** a `Broadcast` followed by another node reports
-  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level, in a
-  branch and in a Router branch. It used to report `TERMINAL_NOT_LAST` with
-  component `Broadcast`.
+  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level and in a
+  branch. It used to report `TERMINAL_NOT_LAST` with component `Broadcast`. In
+  a Router branch a `Broadcast` reports `ROUTER_BRANCH_UNSAFE_NODE`, and one
+  followed by another node also `TERMINAL_NOT_LAST` with component
+  `router branch <label> > Broadcast`.
 - **streaming:** a Router branch that ends in `IntoTopic`, `IntoTable` (any
   `IntoSink`) or `Drain` hands the message to that terminal. The branch's
   terminal was skipped: with a flow `output` the message went to the output
@@ -1346,6 +1392,27 @@
 
 ### streaming
 
+- **streaming:** `ForEach` accepts the batch a `BatchStep` produces, so
+  `CollectBatch → BatchStep → ForEach → IntoTopic` compiles; it failed with
+  `shape mismatch: expected many but got batch before ForEach`. Under
+  at-least-once delivery each offset completes once, after its write is
+  flushed.
+- **streaming:** a batch reaching a node that takes records fails compilation
+  with `SHAPE_MISMATCH`: `IntoTopic`, `IntoTable` (any `IntoSink`), `Fork`,
+  `ExpandRoutes`, `CollectBatch` (it groups records), the flow `output`
+  (component `flow.output`, field `output`)
+  and a Broadcast route `output` (component `broadcast branch <n> > output`,
+  field `output`). Such flows used to compile. `CollectBatch → BatchStep →
+  IntoTopic` raised `'list' object has no attribute 'meta'` at runtime. Put a
+  `ForEach` before the node. `Drain` and `Broadcast` are under Behaviour
+  changes.
+- **streaming:** a `BatchExpandStep` subclass outputs records to the shape
+  check. The compiler matched step output shapes by exact type, so every step
+  subclass kept the incoming shape and a record step after a
+  `BatchExpandStep` failed with `expected record but got batch`. The shape
+  check reads a step's `input_shape` and `output_shape` from its class,
+  except in a Router branch: there a `BatchStep` runs one record at a time,
+  on a one-message batch, so it is record to record.
 - **streaming:** a `Broadcast` is validated as a `Broadcast`. The compiler
   treated it as a plain terminal, so the nodes of its branches were never
   shape-checked. A branch with a shape mismatch, such as a record step after
@@ -1407,6 +1474,14 @@
   `Drain` default failed with `ROUTER_BRANCH_SHAPE_DIVERGENCE`, and a step
   after a `Router` without a default whose branches all end their path failed
   with `SHAPE_MISMATCH`; both compile now.
+- **streaming:** a `BatchExpandStep` expands. Every output had to pair with
+  one input message, so a step returning more or fewer outputs than its batch
+  raised and each message of the batch became an error envelope. Under
+  at-least-once delivery each source offset now completes once per output,
+  after that output's write, and a message with no output completes its
+  offset at the step, also when the batch holds several messages of one
+  offset that an earlier expanding node produced. The new return type is under
+  Behaviour changes.
 
 ### prefect
 

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 7.0.0
+Version: 8.0.1
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/core/backend/sqlalchemy.py

@@ -292,15 +292,19 @@ def _compile_model(
 def _refuse_undeclared_column_types(
     struct_cls: type, column_fields: Mapping[str, ColumnFieldInfo]
 ) -> None:
-    """Refuse a plain enum or ``Literal`` field whose column type is not declared.
+    """Refuse a ``Decimal``, plain enum or ``Literal`` field whose column type is not declared.
 
-    loom does not infer a SQL type from the values of such a field.
+    loom does not infer a SQL type from the values of an enum or a ``Literal``,
+    nor a precision and scale for a ``Decimal``: Postgres rounds a value to the
+    declared scale and refuses one past the declared precision.
     """
     for name, info in column_fields.items():
         if info.needs_declared_type:
             raise ValueError(
-                f"{struct_cls.__name__}.{name}: no column type can be inferred from a plain "
-                "Enum or Literal annotation; declare it, e.g. `ColumnField(String(20))`"
+                f"{struct_cls.__name__}.{name}: no column type is inferred from a Decimal, "
+                "plain Enum or Literal annotation; declare it, e.g. "
+                "`Annotated[Decimal, Numeric(precision=12, scale=2)]` for a Decimal, "
+                "`ColumnField(String(20))` for an Enum or Literal"
             )
 
 

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/core/model/introspection.py

@@ -11,6 +11,7 @@ from typing import (
     Annotated,
     Any,
     ClassVar,
+    Final,
     Literal,
     Union,
     cast,
@@ -56,8 +57,9 @@ class ColumnFieldInfo:
     """Resolved metadata for a single column field.
 
     ``needs_declared_type`` marks a field whose column type was not declared
-    and cannot be inferred, a plain enum or a ``Literal``: ``column_type`` is
-    then a JSON placeholder.
+    and is not inferred: ``column_type`` is then a placeholder, an unbounded
+    ``Numeric`` for a ``Decimal``, whose precision and scale must be declared,
+    and JSON for a plain enum or a ``Literal``.
     """
 
     name: str
@@ -120,7 +122,7 @@ def _resolve_column_field(
         python_type=_extract_origin_type(annotation),
         column_type=column_type or JSON,
         field=field,
-        needs_declared_type=column_type is None,
+        needs_declared_type=column_type is None or column_type is _UNDECLARED_DECIMAL,
     )
 
 
@@ -401,6 +403,10 @@ def _is_classvar(annotation: Any) -> bool:
     return get_origin(annotation) is ClassVar
 
 
+# The placeholder of a Decimal with no declared column type; the SQLAlchemy
+# compiler refuses such a field, so it is never compiled.
+_UNDECLARED_DECIMAL: Final = Numeric()
+
 _SCALAR_TYPE_MAP: dict[type, ColumnType] = {
     int: Integer,
     float: Float,
@@ -409,7 +415,7 @@ _SCALAR_TYPE_MAP: dict[type, ColumnType] = {
     datetime: DateTime(tz=True),
     date: Date,
     time: Time,
-    Decimal: Numeric(),
+    Decimal: _UNDECLARED_DECIMAL,
 }
 
 

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/core/model/types.py

@@ -28,8 +28,16 @@ def DateTime(*, tz: bool = True) -> ColumnType:
     return ColumnType("DateTime", kwargs={"timezone": tz})
 
 
-def Numeric(*, precision: int = 10, scale: int = 2) -> ColumnType:
-    return ColumnType("Numeric", args=(precision, scale))
+def Numeric(*, precision: int | None = None, scale: int | None = None) -> ColumnType:
+    """A ``NUMERIC`` column; with no precision it is unbounded.
+
+    Raises:
+        ValueError: A scale is given without a precision; SQLAlchemy would
+            ignore it.
+    """
+    if scale is not None and precision is None:
+        raise ValueError("Numeric: a scale needs a precision")
+    return ColumnType("Numeric", kwargs={"precision": precision, "scale": scale})
 
 
 class _JsonMarker:

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/streaming/bytewax/handlers/_shared.py

@@ -36,6 +36,7 @@ if TYPE_CHECKING:
 
 Stream: TypeAlias = Any
 AwaitT = TypeVar("AwaitT")
+ExpandT = TypeVar("ExpandT", bound=Iterable[Any])
 
 
 @runtime_checkable
@@ -91,11 +92,8 @@ class _ExecutableBatchExpandStep(Protocol):
         self,
         messages: list[Message[StreamPayload]],
         **kwargs: object,
-    ) -> (
-        Iterable[StreamPayload | Message[StreamPayload]]
-        | Awaitable[Iterable[StreamPayload | Message[StreamPayload]]]
-    ):
-        """Expand one batch into many payloads or replacement messages."""
+    ) -> Iterable[Iterable[StreamPayload]] | Awaitable[Iterable[Iterable[StreamPayload]]]:
+        """Expand each message of one batch into its own list of payloads."""
         ...
 
 
@@ -259,10 +257,9 @@ def _resolve_batch_result(
 
 
 def _resolve_expand_result(
-    result: Iterable[StreamPayload | Message[StreamPayload]]
-    | Awaitable[Iterable[StreamPayload | Message[StreamPayload]]],
+    result: ExpandT | Awaitable[ExpandT],
     node_type: str,
-) -> Iterable[StreamPayload | Message[StreamPayload]]:
+) -> ExpandT:
     """Resolve a synchronous expanding result and reject awaitables."""
     if isinstance(result, Awaitable):
         raise TypeError(f"{node_type} returned an awaitable outside WithAsync.")
@@ -353,9 +350,11 @@ def _reconcile_fanout(
 
     An expanding node turns one input record into N outputs that all carry the
     same source offset, and every one of them completes that offset when it
-    reaches a terminal. The tracker starts each record expecting a single
-    completion, so accounting has to be corrected here — while the outputs are
-    still a plain list and none of them can have reached a terminal yet.
+    reaches a terminal. Each input already owes one completion, and a batch can
+    hold several inputs of one offset when an earlier node expanded it, so per
+    offset the outputs are compared with the inputs: extra outputs fork the
+    difference, and missing ones complete it here — while the outputs are still
+    a plain list and none of them can have reached a terminal yet.
 
     Two failures this prevents, both silent:
 
@@ -372,20 +371,21 @@ def _reconcile_fanout(
     """
     if tracker is None:
         return
-    produced: Counter[tuple[str, int, int]] = Counter()
-    for item in results:
-        key = _commit_key(item)
-        if key is not None:
-            produced[key] += 1
-    for key, count in produced.items():
-        if count > 1:
-            tracker.fork(key[0], key[1], key[2], count - 1)
-    for original in originals:
-        key = _commit_key(original)
-        if key is not None and key not in produced:
+    expected = _commit_key_counts(originals)
+    produced = _commit_key_counts(results)
+    for key in expected | produced:
+        delta = produced[key] - expected[key]
+        if delta > 0:
+            tracker.fork(key[0], key[1], key[2], delta)
+        for _ in range(-delta):
             tracker.complete(key[0], key[1], key[2])
 
 
+def _commit_key_counts(items: Sequence[Any]) -> Counter[tuple[str, int, int]]:
+    """Count the items accountable for each source offset."""
+    return Counter(key for item in items if (key := _commit_key(item)) is not None)
+
+
 def _drop_and_commit(item: Any, tracker: CommitCompletionPort) -> tuple[()]:
     """Drop one item and mark it complete for commit tracking.
 

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/streaming/bytewax/handlers/steps.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from collections.abc import Iterable
+from collections.abc import Iterable, Mapping
 from typing import Any
 
 from bytewax.operators import flat_map
@@ -134,8 +134,6 @@ def _apply_expand_step(
 
     sid = _step_id(f"expand_{idx}_{name}", ctx)
     mapped = bw_map(sid, stream, step_fn)
-    from bytewax.operators import flat_map
-
     flattened_stream = flat_map(_step_id(f"flatten_expand_{idx}", ctx), mapped, _identity)
     return _split_node_result(flattened_stream, sid, ctx, ErrorKind.TASK)
 
@@ -296,9 +294,44 @@ def _execute_batch_expand_step(
         correlation_id=messages[0].meta.correlation_id if messages else None,
     ):
         result = _resolve_expand_result(batch_expand_step.execute(messages), name)
-        if not isinstance(result, Iterable):
-            raise TypeError(f"{name} must return an iterable of payloads.")
-        return _replace_payloads(messages, list(result))
+        return _outputs_per_message(messages, _result_list(result, name), name)
+
+
+def _outputs_per_message(
+    messages: list[Message[StreamPayload]],
+    groups: list[Any],
+    name: str,
+) -> list[Message[StreamPayload]]:
+    """Give each output the metadata of the message whose list it is in.
+
+    ``groups[i]`` holds the outputs of ``messages[i]``; an empty list filters
+    that message out.
+    """
+    if len(groups) != len(messages):
+        raise TypeError(
+            f"{name} returned {len(groups)} output lists for {len(messages)} messages; "
+            "return one list of payloads per message, in batch order."
+        )
+    outputs: list[Message[StreamPayload]] = []
+    for message, payloads in zip(messages, groups, strict=True):
+        for payload in _result_list(payloads, name):
+            if isinstance(payload, Message):
+                raise TypeError(
+                    f"{name} must return payloads, not messages: each output "
+                    "takes the metadata of the message it came from."
+                )
+            outputs.append(_replace_payload(message, payload))
+    return outputs
+
+
+def _result_list(value: object, name: str) -> list[Any]:
+    """Materialize one list of a batch-expanding result, iterating it once.
+
+    ``str``, ``bytes`` and mappings are iterable but are not lists of results.
+    """
+    if not isinstance(value, Iterable) or isinstance(value, (str, bytes, bytearray, Mapping)):
+        raise TypeError(f"{name} must return one list of payloads per message.")
+    return list(value)
 
 
 def _execute_explode_step(

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/streaming/compiler/_plan.py

@@ -31,7 +31,6 @@ class CompiledSingleSource:
     settings: ConsumerSettings
     topics: tuple[str, ...]
     payload_type: type[LoomStruct | LoomFrozenStruct]
-    shape: StreamShape
     decode_strategy: Literal["record", "batch"]
 
 
@@ -45,7 +44,6 @@ class CompiledMultiSource:
         dispatch: Pre-built table mapping ``message_type`` and error
             ``payload_type`` strings to their concrete Python types, plus
             explicit wire-error payload types.
-        shape: Declared source shape.
         decode_strategy: Whether to decode records individually or in batches.
     """
 
@@ -54,7 +52,6 @@ class CompiledMultiSource:
     settings: ConsumerSettings
     topics: tuple[str, ...]
     dispatch: DispatchTable
-    shape: StreamShape
     decode_strategy: Literal["record", "batch"]
 
 
@@ -67,7 +64,6 @@ class CompiledMongoCDCSource:
     settings: MongoSourceConfig
     collections: tuple[str, ...]
     watch_options: Mapping[str, object]
-    shape: StreamShape
 
 
 @dataclass(frozen=True)

{loom_kernel-7.0.0 → loom_kernel-8.0.1}/src/loom/streaming/compiler/phases/build_plan.py

@@ -21,9 +21,8 @@ from loom.streaming.compiler._plan import (
     CompiledStorageSink,
 )
 from loom.streaming.compiler.phases.validate import (
-    _node_needs_async_bridge,
-    _node_output_shape,
-    _walk_all_process_nodes,
+    node_output_shape,
+    walk_process_nodes,
 )
 from loom.streaming.core._errors import ErrorEnvelope
 from loom.streaming.graph._flow import StreamFlow
@@ -33,7 +32,7 @@ from loom.streaming.mongo import MongoConfig
 from loom.streaming.nodes._boundary import FromMultiTypeTopic, FromTopic, IntoTopic
 from loom.streaming.nodes._branches import is_branching_node, iter_branches
 from loom.streaming.nodes._mongo import FromMongoCDC
-from loom.streaming.nodes._shape import CollectBatch
+from loom.streaming.nodes._shape import CollectBatch, StreamShape
 from loom.streaming.nodes._sink import IntoSink
 from loom.streaming.nodes._table import Backend, IntoTable
 from loom.streaming.nodes._table.config import (
@@ -60,7 +59,7 @@ def build_plan(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledPlan:
     terminal_sinks, terminal_storage_sinks = _build_terminal_sinks(flow.process.nodes, ctx)
     error_routes = {kind: _build_sink(topic, ctx) for kind, topic in flow.errors.items()}
     needs_async = any(
-        _node_needs_async_bridge(node) for node in _walk_all_process_nodes(flow.process.nodes)
+        _node_needs_async_bridge(node) for node in walk_process_nodes(flow.process.nodes)
     )
     return CompiledPlan(
         name=flow.name,
@@ -74,6 +73,12 @@ def build_plan(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledPlan:
     )
 
 
+def _node_needs_async_bridge(node: object) -> bool:
+    return isinstance(node, WithAsync) or (
+        isinstance(node, IntoTable) and node.backend is Backend.SQLALCHEMY
+    )
+
+
 def _build_mongo_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledMongoCDCSource:
     source = flow.source
     if not isinstance(source, FromMongoCDC):
@@ -85,7 +90,6 @@ def _build_mongo_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Compi
         settings=settings,
         collections=source.collections,
         watch_options=watch_options,
-        shape=source.shape,
     )
 
 
@@ -100,7 +104,6 @@ def _build_single_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Comp
         settings=consumer,
         topics=consumer.topics,
         payload_type=source.payload,
-        shape=source.shape,
         decode_strategy=decode_strategy,
     )
 
@@ -117,7 +120,6 @@ def _build_multi_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Compi
         settings=consumer,
         topics=consumer.topics,
         dispatch=dispatch,
-        shape=source.shape,
         decode_strategy=decode_strategy,
     )
 
@@ -169,10 +171,10 @@ def _require_message_type(t: type[Any]) -> str:
 
 def _build_nodes(flow: StreamFlow[Any, Any]) -> list[CompiledNode]:
     nodes: list[CompiledNode] = []
-    current_shape = flow.source.shape
+    current_shape = StreamShape.RECORD
     for node in flow.process.nodes:
         input_shape = current_shape
-        output_shape = _node_output_shape(node, current_shape)
+        output_shape = node_output_shape(node, current_shape)
         nodes.append(
             CompiledNode(
                 node=node,