loom-kernel 8.0.0__tar.gz → 8.0.1__tar.gz

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/CHANGELOG.md

@@ -459,6 +459,16 @@
 
 ### core
 
+- **core:** a `Decimal` field, plain or optional, declares its column type: a
+  SQLAlchemy model with an undeclared one is refused when it is compiled,
+  naming the field. loom inferred `NUMERIC(10, 2)`, and Postgres silently
+  rounds a value to that scale (`1.2345` is stored as `1.23`) and refuses one
+  past that precision. Migration: declare it, e.g.
+  `Annotated[Decimal, Numeric(precision=20, scale=6)]` or
+  `ColumnField(Numeric(precision=20, scale=6))`; `Numeric()` with no
+  precision is now an unbounded `NUMERIC` instead of `NUMERIC(10, 2)`, and
+  `Numeric(scale=...)` without a precision raises `ValueError`, as SQLAlchemy
+  would drop the scale.
 - **core:** `sql_endpoint.auth` admits only `identity` and `external`;
   `auth: jwt` is refused when the config is loaded, and the alias and its
   `DeprecationWarning` are gone. Migration: replace `auth: jwt` with
@@ -949,6 +959,22 @@
 
 ### streaming
 
+- **streaming:** `FromTopic`, `FromMultiTypeTopic`, `FromMongoCDC` and
+  `IntoTopic` take no `shape`, `CompiledSingleSource`, `CompiledMultiSource`
+  and `CompiledMongoCDCSource` have no `shape` field, and `StreamShape.MANY`
+  is removed. The runtime never read a declared shape: every source emits one
+  record per message, so a source declared `BATCH` was shape-checked as
+  batches while its nodes received records, and an `IntoTopic` writes one
+  record per message whatever it declared. Every flow now starts as a record
+  stream, and an `IntoTopic` ends its path (shape `none`). Migration: drop
+  `shape=`; to work on batches, use `CollectBatch` and, back to records,
+  `ForEach`.
+- **streaming:** a batch reaching a `Drain` or a `Broadcast` fails compilation
+  with `SHAPE_MISMATCH`, whatever the delivery. Under at-most-once delivery
+  `CollectBatch → Drain` compiled and dropped the batch; under at-least-once
+  the batch's offsets never completed. Under at-least-once a `Broadcast` with
+  more than one route raised `TypeError` (`Expected Message, got list`) on a
+  batch. Put a `ForEach` before the node.
 - **streaming:** `BatchExpandStep.execute` returns one list of payloads per
   message of its batch, in batch order, instead of one flat list. List `i`
   holds what `messages[i]` expands into; each output takes that message's
@@ -975,9 +1001,11 @@
   `CompileIssue` instead), `loom_message_type()` is gone, and
   `CompilationIssue.prefixed()` is removed. Messages keep their text.
 - **streaming:** a `Broadcast` followed by another node reports
-  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level, in a
-  branch and in a Router branch. It used to report `TERMINAL_NOT_LAST` with
-  component `Broadcast`.
+  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level and in a
+  branch. It used to report `TERMINAL_NOT_LAST` with component `Broadcast`. In
+  a Router branch a `Broadcast` reports `ROUTER_BRANCH_UNSAFE_NODE`, and one
+  followed by another node also `TERMINAL_NOT_LAST` with component
+  `router branch <label> > Broadcast`.
 - **streaming:** a Router branch that ends in `IntoTopic`, `IntoTable` (any
   `IntoSink`) or `Drain` hands the message to that terminal. The branch's
   terminal was skipped: with a flow `output` the message went to the output
@@ -1364,6 +1392,27 @@
 
 ### streaming
 
+- **streaming:** `ForEach` accepts the batch a `BatchStep` produces, so
+  `CollectBatch → BatchStep → ForEach → IntoTopic` compiles; it failed with
+  `shape mismatch: expected many but got batch before ForEach`. Under
+  at-least-once delivery each offset completes once, after its write is
+  flushed.
+- **streaming:** a batch reaching a node that takes records fails compilation
+  with `SHAPE_MISMATCH`: `IntoTopic`, `IntoTable` (any `IntoSink`), `Fork`,
+  `ExpandRoutes`, `CollectBatch` (it groups records), the flow `output`
+  (component `flow.output`, field `output`)
+  and a Broadcast route `output` (component `broadcast branch <n> > output`,
+  field `output`). Such flows used to compile. `CollectBatch → BatchStep →
+  IntoTopic` raised `'list' object has no attribute 'meta'` at runtime. Put a
+  `ForEach` before the node. `Drain` and `Broadcast` are under Behaviour
+  changes.
+- **streaming:** a `BatchExpandStep` subclass outputs records to the shape
+  check. The compiler matched step output shapes by exact type, so every step
+  subclass kept the incoming shape and a record step after a
+  `BatchExpandStep` failed with `expected record but got batch`. The shape
+  check reads a step's `input_shape` and `output_shape` from its class,
+  except in a Router branch: there a `BatchStep` runs one record at a time,
+  on a one-message batch, so it is record to record.
 - **streaming:** a `Broadcast` is validated as a `Broadcast`. The compiler
   treated it as a plain terminal, so the nodes of its branches were never
   shape-checked. A branch with a shape mismatch, such as a record step after

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 8.0.0
+Version: 8.0.1
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/core/backend/sqlalchemy.py

@@ -292,15 +292,19 @@ def _compile_model(
 def _refuse_undeclared_column_types(
     struct_cls: type, column_fields: Mapping[str, ColumnFieldInfo]
 ) -> None:
-    """Refuse a plain enum or ``Literal`` field whose column type is not declared.
+    """Refuse a ``Decimal``, plain enum or ``Literal`` field whose column type is not declared.
 
-    loom does not infer a SQL type from the values of such a field.
+    loom does not infer a SQL type from the values of an enum or a ``Literal``,
+    nor a precision and scale for a ``Decimal``: Postgres rounds a value to the
+    declared scale and refuses one past the declared precision.
     """
     for name, info in column_fields.items():
         if info.needs_declared_type:
             raise ValueError(
-                f"{struct_cls.__name__}.{name}: no column type can be inferred from a plain "
-                "Enum or Literal annotation; declare it, e.g. `ColumnField(String(20))`"
+                f"{struct_cls.__name__}.{name}: no column type is inferred from a Decimal, "
+                "plain Enum or Literal annotation; declare it, e.g. "
+                "`Annotated[Decimal, Numeric(precision=12, scale=2)]` for a Decimal, "
+                "`ColumnField(String(20))` for an Enum or Literal"
             )
 
 

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/core/model/introspection.py

@@ -11,6 +11,7 @@ from typing import (
     Annotated,
     Any,
     ClassVar,
+    Final,
     Literal,
     Union,
     cast,
@@ -56,8 +57,9 @@ class ColumnFieldInfo:
     """Resolved metadata for a single column field.
 
     ``needs_declared_type`` marks a field whose column type was not declared
-    and cannot be inferred, a plain enum or a ``Literal``: ``column_type`` is
-    then a JSON placeholder.
+    and is not inferred: ``column_type`` is then a placeholder, an unbounded
+    ``Numeric`` for a ``Decimal``, whose precision and scale must be declared,
+    and JSON for a plain enum or a ``Literal``.
     """
 
     name: str
@@ -120,7 +122,7 @@ def _resolve_column_field(
         python_type=_extract_origin_type(annotation),
         column_type=column_type or JSON,
         field=field,
-        needs_declared_type=column_type is None,
+        needs_declared_type=column_type is None or column_type is _UNDECLARED_DECIMAL,
     )
 
 
@@ -401,6 +403,10 @@ def _is_classvar(annotation: Any) -> bool:
     return get_origin(annotation) is ClassVar
 
 
+# The placeholder of a Decimal with no declared column type; the SQLAlchemy
+# compiler refuses such a field, so it is never compiled.
+_UNDECLARED_DECIMAL: Final = Numeric()
+
 _SCALAR_TYPE_MAP: dict[type, ColumnType] = {
     int: Integer,
     float: Float,
@@ -409,7 +415,7 @@ _SCALAR_TYPE_MAP: dict[type, ColumnType] = {
     datetime: DateTime(tz=True),
     date: Date,
     time: Time,
-    Decimal: Numeric(),
+    Decimal: _UNDECLARED_DECIMAL,
 }
 
 

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/core/model/types.py

@@ -28,8 +28,16 @@ def DateTime(*, tz: bool = True) -> ColumnType:
     return ColumnType("DateTime", kwargs={"timezone": tz})
 
 
-def Numeric(*, precision: int = 10, scale: int = 2) -> ColumnType:
-    return ColumnType("Numeric", args=(precision, scale))
+def Numeric(*, precision: int | None = None, scale: int | None = None) -> ColumnType:
+    """A ``NUMERIC`` column; with no precision it is unbounded.
+
+    Raises:
+        ValueError: A scale is given without a precision; SQLAlchemy would
+            ignore it.
+    """
+    if scale is not None and precision is None:
+        raise ValueError("Numeric: a scale needs a precision")
+    return ColumnType("Numeric", kwargs={"precision": precision, "scale": scale})
 
 
 class _JsonMarker:

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/compiler/_plan.py

@@ -31,7 +31,6 @@ class CompiledSingleSource:
     settings: ConsumerSettings
     topics: tuple[str, ...]
     payload_type: type[LoomStruct | LoomFrozenStruct]
-    shape: StreamShape
     decode_strategy: Literal["record", "batch"]
 
 
@@ -45,7 +44,6 @@ class CompiledMultiSource:
         dispatch: Pre-built table mapping ``message_type`` and error
             ``payload_type`` strings to their concrete Python types, plus
             explicit wire-error payload types.
-        shape: Declared source shape.
         decode_strategy: Whether to decode records individually or in batches.
     """
 
@@ -54,7 +52,6 @@ class CompiledMultiSource:
     settings: ConsumerSettings
     topics: tuple[str, ...]
     dispatch: DispatchTable
-    shape: StreamShape
     decode_strategy: Literal["record", "batch"]
 
 
@@ -67,7 +64,6 @@ class CompiledMongoCDCSource:
     settings: MongoSourceConfig
     collections: tuple[str, ...]
     watch_options: Mapping[str, object]
-    shape: StreamShape
 
 
 @dataclass(frozen=True)

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/compiler/phases/build_plan.py

@@ -21,9 +21,8 @@ from loom.streaming.compiler._plan import (
     CompiledStorageSink,
 )
 from loom.streaming.compiler.phases.validate import (
-    _node_needs_async_bridge,
-    _node_output_shape,
-    _walk_all_process_nodes,
+    node_output_shape,
+    walk_process_nodes,
 )
 from loom.streaming.core._errors import ErrorEnvelope
 from loom.streaming.graph._flow import StreamFlow
@@ -33,7 +32,7 @@ from loom.streaming.mongo import MongoConfig
 from loom.streaming.nodes._boundary import FromMultiTypeTopic, FromTopic, IntoTopic
 from loom.streaming.nodes._branches import is_branching_node, iter_branches
 from loom.streaming.nodes._mongo import FromMongoCDC
-from loom.streaming.nodes._shape import CollectBatch
+from loom.streaming.nodes._shape import CollectBatch, StreamShape
 from loom.streaming.nodes._sink import IntoSink
 from loom.streaming.nodes._table import Backend, IntoTable
 from loom.streaming.nodes._table.config import (
@@ -60,7 +59,7 @@ def build_plan(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledPlan:
     terminal_sinks, terminal_storage_sinks = _build_terminal_sinks(flow.process.nodes, ctx)
     error_routes = {kind: _build_sink(topic, ctx) for kind, topic in flow.errors.items()}
     needs_async = any(
-        _node_needs_async_bridge(node) for node in _walk_all_process_nodes(flow.process.nodes)
+        _node_needs_async_bridge(node) for node in walk_process_nodes(flow.process.nodes)
     )
     return CompiledPlan(
         name=flow.name,
@@ -74,6 +73,12 @@ def build_plan(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledPlan:
     )
 
 
+def _node_needs_async_bridge(node: object) -> bool:
+    return isinstance(node, WithAsync) or (
+        isinstance(node, IntoTable) and node.backend is Backend.SQLALCHEMY
+    )
+
+
 def _build_mongo_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> CompiledMongoCDCSource:
     source = flow.source
     if not isinstance(source, FromMongoCDC):
@@ -85,7 +90,6 @@ def _build_mongo_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Compi
         settings=settings,
         collections=source.collections,
         watch_options=watch_options,
-        shape=source.shape,
     )
 
 
@@ -100,7 +104,6 @@ def _build_single_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Comp
         settings=consumer,
         topics=consumer.topics,
         payload_type=source.payload,
-        shape=source.shape,
         decode_strategy=decode_strategy,
     )
 
@@ -117,7 +120,6 @@ def _build_multi_source(flow: StreamFlow[Any, Any], ctx: ConfigContext) -> Compi
         settings=consumer,
         topics=consumer.topics,
         dispatch=dispatch,
-        shape=source.shape,
         decode_strategy=decode_strategy,
     )
 
@@ -169,10 +171,10 @@ def _require_message_type(t: type[Any]) -> str:
 
 def _build_nodes(flow: StreamFlow[Any, Any]) -> list[CompiledNode]:
     nodes: list[CompiledNode] = []
-    current_shape = flow.source.shape
+    current_shape = StreamShape.RECORD
     for node in flow.process.nodes:
         input_shape = current_shape
-        output_shape = _node_output_shape(node, current_shape)
+        output_shape = node_output_shape(node, current_shape)
         nodes.append(
             CompiledNode(
                 node=node,

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/compiler/phases/validate.py

@@ -36,7 +36,7 @@ from loom.streaming.nodes._mongo import FromMongoCDC
 from loom.streaming.nodes._router import Router
 from loom.streaming.nodes._shape import CollectBatch, Drain, ForEach, StreamShape, WindowStrategy
 from loom.streaming.nodes._sink import IntoSink
-from loom.streaming.nodes._step import BatchExpandStep, BatchStep, ExpandStep, RecordStep
+from loom.streaming.nodes._step import BatchExpandStep, BatchStep, ExpandStep, RecordStep, Step
 from loom.streaming.nodes._table import Backend, IntoTable
 from loom.streaming.nodes._table.config import (
     resolve_clickhouse_table_config,
@@ -119,7 +119,9 @@ def validate_resources(flow: StreamFlow[Any, Any]) -> list[CompilationIssue]:
 
 def validate_shapes(flow: StreamFlow[Any, Any]) -> list[CompilationIssue]:
     """Validate shape transitions through the process tree."""
-    errors, _ = _validate_shape_sequence(flow.process.nodes, flow.source.shape)
+    errors, final_shape = _validate_shape_sequence(flow.process.nodes, StreamShape.RECORD)
+    if flow.output is not None:
+        errors.extend(_output_shape_issues(final_shape, "flow.output"))
     errors.extend(_validate_window_strategies(flow.process.nodes))
     errors.extend(_validate_scoped_process_nodes(_walk_all_process_nodes(flow.process.nodes)))
     return errors
@@ -224,18 +226,12 @@ def walk_process_nodes(nodes: Iterable[object]) -> Iterable[object]:
     yield from _walk_all_process_nodes(nodes)
 
 
-def _node_needs_async_bridge(node: object) -> bool:
-    return isinstance(node, WithAsync) or (
-        isinstance(node, IntoTable) and node.backend is Backend.SQLALCHEMY
-    )
-
-
 def _check_input_shape(
     node: object,
     current_shape: StreamShape,
     errors: list[CompilationIssue],
 ) -> None:
-    expected = _node_input_shape(node)
+    expected = node_input_shape(node)
     if expected is not None and current_shape != expected:
         errors.append(
             StreamingErrorCode.SHAPE_MISMATCH.issue(
@@ -244,6 +240,21 @@ def _check_input_shape(
         )
 
 
+def _output_shape_issues(shape: StreamShape, component: str) -> list[CompilationIssue]:
+    """Return the mismatch of an output declared after a stream of *shape*.
+
+    An output writes records, so only a ``batch`` is a mismatch. ``none`` is the
+    shape after a node that ends the path, and after ``With`` and ``WithAsync``,
+    whose runtime hands their results on one message at a time.
+    """
+    if shape in (StreamShape.RECORD, StreamShape.NONE):
+        return []
+    issue = StreamingErrorCode.SHAPE_MISMATCH.issue(
+        expected=StreamShape.RECORD.value, got=shape.value, component=component
+    )
+    return [msgspec.structs.replace(issue, field="output")]
+
+
 def _must_be_last_errors(
     idx: int,
     node_list: tuple[object, ...],
@@ -351,13 +362,13 @@ def _validate_shape_node(
     if isinstance(node, ExpandRoutes):
         return _validate_expand_routes_node(node, idx, node_list)
     if isinstance(node, Router):
-        router_errors, next_shape = _validate_router_shapes(node, StreamShape.RECORD)
+        router_errors, next_shape = _validate_router_shapes(node)
         if router_ends_every_path(node):
             not_last = StreamingErrorCode.TERMINAL_NOT_LAST.issue(component=type(node).__name__)
             router_errors.extend(_must_be_last_errors(idx, node_list, not_last))
             return router_errors, StreamShape.NONE, True
         return router_errors, next_shape, False
-    return [], _node_output_shape(node, current_shape), False
+    return [], node_output_shape(node, current_shape), False
 
 
 def _validate_leaf_terminal_node(
@@ -421,7 +432,7 @@ def _validate_explode_node(
     if not isinstance(next_node, Router):
         got = type(next_node).__name__ if next_node is not None else "nothing"
         errors.append(StreamingErrorCode.EXPLODE_WITHOUT_ROUTER.issue(got=got))
-    return errors, _node_output_shape(node, current_shape), False
+    return errors, node_output_shape(node, current_shape), False
 
 
 def _prefixed(issue: CompilationIssue, prefix: str) -> CompilationIssue:
@@ -471,7 +482,9 @@ def _validate_broadcast_shapes(
 ) -> tuple[list[CompilationIssue], StreamShape]:
     errors: list[CompilationIssue] = []
     for branch in iter_branches(broadcast):
-        branch_errors, _ = _validate_shape_sequence(branch.nodes, initial_shape)
+        branch_errors, final_shape = _validate_shape_sequence(branch.nodes, initial_shape)
+        if branch.output is not None:
+            branch_errors.extend(_output_shape_issues(final_shape, "output"))
         errors.extend(
             _prefixed(issue, f"broadcast branch {branch.label}") for issue in branch_errors
         )
@@ -487,50 +500,47 @@ def _validate_broadcast_shapes(
 
 def _validate_router_branch_shape_sequence(
     nodes: Iterable[object],
-    initial_shape: StreamShape,
 ) -> tuple[list[CompilationIssue], StreamShape]:
+    """Shape-check the nodes of one Router branch and return the branch's output shape.
+
+    A ``BatchStep`` here is record to record: the runtime runs it on a
+    one-message batch of the routed message
+    (``loom.streaming.bytewax.handlers.routing._execute_router_node``).
+    """
     errors: list[CompilationIssue] = []
-    current_shape = initial_shape
+    current_shape = StreamShape.RECORD
     node_list = tuple(nodes)
 
     for idx, node in enumerate(node_list):
         if isinstance(node, BatchStep):
             current_shape = StreamShape.RECORD
             continue
-        expected = _node_input_shape(node)
-        if expected is not None and current_shape != expected:
-            errors.append(
-                StreamingErrorCode.SHAPE_MISMATCH.issue(
-                    expected=expected.value, got=current_shape.value, component=type(node).__name__
-                )
-            )
+        _check_input_shape(node, current_shape, errors)
         if _is_leaf_terminal(node) and idx != len(node_list) - 1:
-            errors.append(
-                StreamingErrorCode.BROADCAST_NOT_LAST.issue()
-                if isinstance(node, Broadcast)
-                else StreamingErrorCode.TERMINAL_NOT_LAST.issue(component=type(node).__name__)
-            )
+            errors.append(StreamingErrorCode.TERMINAL_NOT_LAST.issue(component=type(node).__name__))
             break
-        current_shape = _node_output_shape(node, current_shape)
+        current_shape = node_output_shape(node, current_shape)
 
     return errors, current_shape
 
 
 def _validate_router_shapes(
     router: Router[StreamPayload, StreamPayload],
-    initial_shape: StreamShape,
 ) -> tuple[list[CompilationIssue], StreamShape]:
     """Validate *router*'s branches and return the shape of the messages that continue.
 
     Those are the outputs of the branches that do not end their path and, without
     a default branch, the unmatched input; only they must agree on one shape.
+    A Router takes records, so every branch starts as a record stream.
     """
     errors: list[CompilationIssue] = []
-    outputs: dict[StreamShape, None] = {} if router.default is not None else {initial_shape: None}
+    outputs: dict[StreamShape, None] = (
+        {} if router.default is not None else {StreamShape.RECORD: None}
+    )
 
     for branch in iter_branches(router):
         label, nodes = branch.label, branch.nodes
-        branch_errors, branch_output = _validate_router_branch_shape_sequence(nodes, initial_shape)
+        branch_errors, branch_output = _validate_router_branch_shape_sequence(nodes)
         errors.extend(_prefixed(issue, f"router branch {label}") for issue in branch_errors)
         for node in nodes:
             if not isinstance(node, RouterBranchSafe):
@@ -644,30 +654,6 @@ def _contains_broadcast(nodes: Iterable[object]) -> bool:
     return any(isinstance(node, Broadcast) for node in nodes)
 
 
-def _node_input_shape(node: object) -> StreamShape | None:
-    if isinstance(node, RecordStep):
-        return StreamShape.RECORD
-    if isinstance(node, BatchStep):
-        return StreamShape.BATCH
-    if isinstance(node, ExpandStep):
-        return StreamShape.RECORD
-    if isinstance(node, BatchExpandStep):
-        return StreamShape.BATCH
-    if isinstance(node, Explode):
-        return StreamShape.RECORD
-    if isinstance(node, (With, WithAsync)):
-        return None
-    if isinstance(node, ForEach):
-        return StreamShape.MANY
-    if isinstance(node, Drain):
-        return None
-    if isinstance(node, Fork):
-        return None
-    if isinstance(node, Router):
-        return StreamShape.RECORD
-    return None
-
-
 def _validate_window_strategies(nodes: Iterable[object]) -> list[CompilationIssue]:
     errors: list[CompilationIssue] = []
     for node in nodes:
@@ -702,29 +688,47 @@ def _validate_scoped_process_nodes(nodes: Iterable[object]) -> list[CompilationI
     return errors
 
 
-_FIXED_OUTPUT_SHAPES: dict[type, StreamShape] = {
-    CollectBatch: StreamShape.BATCH,
-    Explode: StreamShape.RECORD,
-    ForEach: StreamShape.RECORD,
-    RecordStep: StreamShape.RECORD,
-    BatchStep: StreamShape.BATCH,
-    ExpandStep: StreamShape.RECORD,
-    BatchExpandStep: StreamShape.RECORD,
-    WithAsync: StreamShape.NONE,
-    Drain: StreamShape.NONE,
-    Fork: StreamShape.NONE,
-    Broadcast: StreamShape.NONE,
-}
-
-
-def _node_output_shape(node: object, current: StreamShape) -> StreamShape:
-    fixed = _FIXED_OUTPUT_SHAPES.get(type(node))
-    if fixed is not None:
-        return fixed
-    if isinstance(node, IntoSink):
-        return StreamShape.NONE
-    if isinstance(node, IntoTopic):
-        return node.shape
-    if isinstance(node, With):
-        return StreamShape.NONE if node.process is not None else StreamShape.MANY
-    return current
+_NODE_SHAPES: tuple[tuple[type, StreamShape | None, StreamShape | None], ...] = (
+    (CollectBatch, StreamShape.RECORD, StreamShape.BATCH),
+    (ForEach, StreamShape.BATCH, StreamShape.RECORD),
+    (Explode, StreamShape.RECORD, None),
+    (Router, StreamShape.RECORD, None),
+    (ExpandRoutes, StreamShape.RECORD, None),
+    (Fork, StreamShape.RECORD, StreamShape.NONE),
+    (Broadcast, StreamShape.RECORD, StreamShape.NONE),
+    (Drain, StreamShape.RECORD, StreamShape.NONE),
+    (IntoTopic, StreamShape.RECORD, StreamShape.NONE),
+    (IntoSink, StreamShape.RECORD, StreamShape.NONE),
+    (With, None, StreamShape.NONE),
+    (WithAsync, None, StreamShape.NONE),
+)
+"""``(node type, input shape, output shape)`` of every non-step node.
+
+``None`` input accepts any shape; ``None`` output keeps the incoming shape.
+A ``Step`` declares both on its class instead.
+"""
+
+
+def _non_step_shapes(node: object) -> tuple[StreamShape | None, StreamShape | None]:
+    return next(
+        ((inp, out) for kind, inp, out in _NODE_SHAPES if isinstance(node, kind)), (None, None)
+    )
+
+
+def node_input_shape(node: object) -> StreamShape | None:
+    """Return the shape *node* requires on its input, or ``None`` when it takes any."""
+    if isinstance(node, Step):
+        return node.input_shape
+    return _non_step_shapes(node)[0]
+
+
+def node_output_shape(node: object, current: StreamShape) -> StreamShape:
+    """Return the shape after *node* when it receives *current*.
+
+    In a Router branch a ``BatchStep`` is record to record; the branch walk
+    handles it before calling this.
+    """
+    if isinstance(node, Step):
+        return node.output_shape
+    output = _non_step_shapes(node)[1]
+    return current if output is None else output

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/nodes/_boundary.py

@@ -8,7 +8,6 @@ from typing import ClassVar, Generic, Protocol, TypeVar
 from loom.core.model import LoomFrozenStruct, LoomStruct
 from loom.core.routing import LogicalRef, as_logical_ref
 from loom.streaming.core._message import Message
-from loom.streaming.nodes._shape import StreamShape
 
 PayloadT = TypeVar("PayloadT", bound=LoomStruct | LoomFrozenStruct, contravariant=True)
 MultiPayloadT = TypeVar("MultiPayloadT", bound=LoomStruct | LoomFrozenStruct, covariant=True)
@@ -24,12 +23,10 @@ class FromTopic(LoomFrozenStruct, Generic[PayloadT], frozen=True):
         name: Logical input reference. By default this is also used as the
             Kafka config reference.
         payload: Logical payload type descriptor.
-        shape: Declared source shape.
     """
 
     name: str
     payload: type[PayloadT]
-    shape: StreamShape = StreamShape.RECORD
 
     @property
     def logical_ref(self) -> LogicalRef:
@@ -56,7 +53,6 @@ class FromMultiTypeTopic(LoomFrozenStruct, Generic[MultiPayloadT], frozen=True):
     Args:
         name: Logical input reference.
         payloads: Tuple of two or more expected payload types.
-        shape: Declared source shape.
 
     Raises:
         ValueError: When fewer than two payload types are declared.
@@ -72,7 +68,6 @@ class FromMultiTypeTopic(LoomFrozenStruct, Generic[MultiPayloadT], frozen=True):
 
     name: str
     payloads: tuple[type[LoomStruct | LoomFrozenStruct], ...]
-    shape: StreamShape = StreamShape.RECORD
 
     def __post_init__(self) -> None:
         if len(self.payloads) < 2:
@@ -97,7 +92,6 @@ class IntoTopic(LoomFrozenStruct, Generic[PayloadT], frozen=True):
         name: Logical output reference. By default this is also used as the
             Kafka config reference.
         payload: Optional logical payload type descriptor.
-        shape: Declared output shape.
         partitioning: Optional partitioning policy.
         dlq: Optional dead-letter topic name.  When set, messages that fail
             delivery are routed to this topic instead of crashing the worker.
@@ -107,7 +101,6 @@ class IntoTopic(LoomFrozenStruct, Generic[PayloadT], frozen=True):
 
     name: str
     payload: type[PayloadT] | None = None
-    shape: StreamShape = StreamShape.RECORD
     partitioning: PartitionPolicy[PayloadT] | None = None
     dlq: str | None = None
     router_branch_safe: ClassVar[bool] = True

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/nodes/_mongo.py

@@ -11,7 +11,6 @@ from typing_extensions import TypeVar
 from loom.core.model import LoomFrozenStruct, LoomStruct
 from loom.core.routing import LogicalRef, as_logical_ref
 from loom.streaming.mongo._event import MongoCDCEvent
-from loom.streaming.nodes._shape import StreamShape
 
 MongoCDCPayloadT = TypeVar(
     "MongoCDCPayloadT",
@@ -33,13 +32,11 @@ class FromMongoCDC(LoomFrozenStruct, Generic[MongoCDCPayloadT], frozen=True):
         collections: Collection names to watch. An empty tuple means a
             database-level watch.
         watch_options: Driver watch kwargs resolved later by compiler/runtime.
-        shape: Declared source shape.
     """
 
     name: str
     collections: tuple[str, ...] = ()
     watch_options: Mapping[str, object] = msgspec.field(default_factory=dict)
-    shape: StreamShape = StreamShape.RECORD
 
     @property
     def logical_ref(self) -> LogicalRef:

{loom_kernel-8.0.0 → loom_kernel-8.0.1}/src/loom/streaming/nodes/_shape.py

@@ -12,7 +12,6 @@ class StreamShape(StrEnum):
     """Logical data shape at a streaming graph edge."""
 
     RECORD = "record"
-    MANY = "many"
     BATCH = "batch"
     NONE = "none"
 
@@ -64,8 +63,8 @@ class CollectBatch(LoomFrozenStruct, frozen=True):
 
     Groups individual records into batches before handing them to downstream
     batch-aware nodes. Batch is an **optimization grouping**, not a
-    first-class semantic unit — downstream nodes receive and return individual
-    records after the batch step executes.
+    first-class semantic unit: a ``ForEach`` after the batch nodes turns the
+    batch back into individual records.
 
     Args:
         max_records: Maximum records collected into one batch.
@@ -102,7 +101,7 @@ class CollectBatch(LoomFrozenStruct, frozen=True):
 
 
 class Drain(LoomFrozenStruct, frozen=True):
-    """Explicit terminal adapter from any shape to ``none``.
+    """Explicit terminal adapter from ``record`` to ``none``.
 
     Pattern:
         Shape adapter.