loom-kernel 2.15.0__tar.gz → 2.16.0__tar.gz

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/CHANGELOG.md

@@ -225,6 +225,11 @@
   downstream run inherits the placeholder text and resolves it against its own
   creation time. Without `trigger` the deployment is registered exactly as
   before.
+- **prefect:** `FlowTrigger`, the decoded `trigger` block, is exported from
+  `loom.prefect` and `loom.prefect.flow`, and `check_inherited_params(trigger,
+  params_type, *, name)` from `loom.prefect.flow`, so CI can check that every
+  inherited parameter is a field declared `str`, `date` or `datetime` without
+  building the flow.
 
 ### etl
 
@@ -278,6 +283,36 @@
   `MISSING_CONFIG_PARAMS`, `CONFIG_ALIAS_CONFLICT` and
   `UNSUPPORTED_CONFIG_VALUE`. `ETLRunner`, `ETLRunner.from_config`,
   `ETLCompiler` and `ETLExecutor` take `agents=`. Polars only.
+- **etl:** `loom.etl.compiler.filter_plan(plan, include)` returns the plan
+  `ETLRunner.run(..., include=...)` executes: a process name keeps the whole
+  process, a step name keeps that step, and a name that is neither raises
+  `InvalidStageError`, listing the unknown names sorted, so a deployment's
+  selection can be checked without running it. A bare string as `include`
+  raises `TypeError`. The runner now filters through it. `InvalidStageError`
+  is exported from `loom.etl.compiler` as well; it is the same class as
+  `loom.etl.runner.InvalidStageError` and `loom.etl.InvalidStageError`.
+- **etl:** `iter_processes`, `iter_steps_in_process` and `iter_all_steps` are
+  exported from `loom.etl.compiler`. They flatten a compiled plan in
+  execution order, parallel groups in declaration order, so code that walks
+  a plan no longer reimplements them over the private `_plan` module.
+- **etl:** `loom.etl.backends.polars.decode_json_columns(frame, json_columns)`
+  decodes JSON text columns exactly as the Polars reader does for a source
+  declared with `parse_json`, normalising fields whose JSON type changes
+  between rows, and returns a frame of the same kind, eager or lazy. The
+  reader decodes through it. `FromTable.json_columns` and
+  `FromFile.json_columns` expose the declarations it takes, so a test can
+  decode a seed as the step would receive it. Spark keeps its `from_json`
+  decoding, which has no normalisation to share.
+- **etl:** `IntoTable(...).replace(allow_empty=False)` refuses to wipe a table
+  with an empty frame: the writer raises `EmptyReplaceError` (a `ValueError`,
+  exported from `loom.etl`, carrying `table_ref`) before it creates or writes
+  anything, so a dimension rebuilt from a read that came back empty keeps its
+  rows and the step fails. Polars checks the row count of the frame the
+  replace already collects, at no extra cost; Spark runs `isEmpty()`, one
+  job that stops at the first row, before the write recomputes the frame. On
+  Polars a `Null`-typed column still fails first, with `SchemaError`.
+  The default, `allow_empty=True`, keeps today's behaviour, and `ReplaceSpec`
+  gains the `allow_empty` field.
 
 ## ⚠ Behaviour changes
 
@@ -368,6 +403,14 @@
   which is harmless. The update is one attempt with a 10-second timeout; if
   it fails, the flow logs it at `ERROR` in the run's logs and goes on.
 
+### etl
+
+- **etl:** the names of `loom.etl.runner.filtering` (`_filter_plan`,
+  `_filter_process`, `_collect_all_names`, `_map_pipeline_nodes`,
+  `_map_process_nodes`) are deprecated and removed in loom 3.0; each emits a
+  `DeprecationWarning` naming its replacement, `loom.etl.compiler.filter_plan`
+  or, for `_collect_all_names`, `iter_processes` and `iter_all_steps`.
+
 ## 🐛 Fixes
 
 ### core
@@ -430,6 +473,19 @@
 
 ### ai
 
+- **ai:** a configured `ai.prices` rate prices runs again with pydantic-ai
+  2.54. That release records a response's usage before the
+  `after_model_request` hook loom priced it in, so every run reported
+  `AgentUsage.cost=None`: `policies.max_usd` stopped binding, a reported cost
+  was summed instead of replaced, and the ETL budget ledger kept each run's
+  whole reservation. The price is now applied by a model wrapper as the model
+  returns each response, in both run modes, before any engine release records
+  it. The lock moves to pydantic-ai-slim 2.54.0, which sends `tool_choice:
+  auto` for a `tool` output with adaptive thinking, so Anthropic models reason
+  again (2.47 forced the tool). `UsageExtractionFailedWarning`, new in 2.52, is
+  classified `COST_NOT_MEASURABLE` like its siblings. Every `ai-*` extra now
+  declares `pydantic-ai-slim>=2.45`, the oldest release loom imports and passes
+  its agent suites with, and CI runs those suites against that floor.
 - **ai:** an MCP session that dies after it opened no longer fails every later
   call until the worker restarts. The worker holds one session per server for
   its whole life, and that holder kept the dead client from ever being

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 2.15.0
+Version: 2.16.0
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3
@@ -22,17 +22,17 @@ Provides-Extra: ai-a2a
 Requires-Dist: a2a-sdk<2,>=1.1; extra == 'ai-a2a'
 Requires-Dist: fasta2a<1,>=0.6; extra == 'ai-a2a'
 Provides-Extra: ai-anthropic
-Requires-Dist: pydantic-ai-slim[anthropic]<3,>=2; extra == 'ai-anthropic'
+Requires-Dist: pydantic-ai-slim[anthropic]<3,>=2.45; extra == 'ai-anthropic'
 Provides-Extra: ai-bedrock
-Requires-Dist: pydantic-ai-slim[bedrock]<3,>=2; extra == 'ai-bedrock'
+Requires-Dist: pydantic-ai-slim[bedrock]<3,>=2.45; extra == 'ai-bedrock'
 Provides-Extra: ai-harness
 Requires-Dist: pydantic-ai-harness[skills]<1,>=0.28; extra == 'ai-harness'
 Provides-Extra: ai-openai
-Requires-Dist: pydantic-ai-slim[openai]<3,>=2; extra == 'ai-openai'
+Requires-Dist: pydantic-ai-slim[openai]<3,>=2.45; extra == 'ai-openai'
 Provides-Extra: ai-pydantic
-Requires-Dist: pydantic-ai-slim<3,>=2; extra == 'ai-pydantic'
+Requires-Dist: pydantic-ai-slim<3,>=2.45; extra == 'ai-pydantic'
 Provides-Extra: ai-templates
-Requires-Dist: pydantic-ai-slim[spec]<3,>=2; extra == 'ai-templates'
+Requires-Dist: pydantic-ai-slim[spec]<3,>=2.45; extra == 'ai-templates'
 Provides-Extra: ai-typesafe
 Requires-Dist: pydantic-ai-slim[typesafe]<3,>=2.45; extra == 'ai-typesafe'
 Provides-Extra: cache

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/pyproject.toml

@@ -124,16 +124,16 @@ streaming = [
     "opentelemetry-exporter-otlp-proto-grpc>=1.27,<2.0",
 ]
 ai-pydantic = [
-    "pydantic-ai-slim>=2,<3",
+    "pydantic-ai-slim>=2.45,<3",
 ]
 ai-bedrock = [
-    "pydantic-ai-slim[bedrock]>=2,<3",
+    "pydantic-ai-slim[bedrock]>=2.45,<3",
 ]
 ai-openai = [
-    "pydantic-ai-slim[openai]>=2,<3",
+    "pydantic-ai-slim[openai]>=2.45,<3",
 ]
 ai-anthropic = [
-    "pydantic-ai-slim[anthropic]>=2,<3",
+    "pydantic-ai-slim[anthropic]>=2.45,<3",
 ]
 # TypeSafe's Jev: a decision model that fills a typed output from a state, no
 # text. Its model class first shipped in pydantic-ai 2.45.
@@ -144,7 +144,7 @@ ai-typesafe = [
 # the engine compiles a Handlebars template and renders it per request. An
 # artefact whose instructions are literal never reaches the import.
 ai-templates = [
-    "pydantic-ai-slim[spec]>=2,<3",
+    "pydantic-ai-slim[spec]>=2.45,<3",
 ]
 ai-a2a = [
     "fasta2a>=0.6,<1",

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/ai/engines/pydantic_ai/_errors.py

@@ -16,6 +16,7 @@ from __future__ import annotations
 from collections.abc import Mapping
 from types import MappingProxyType
 
+from pydantic_ai import exceptions as pydantic_ai_exceptions
 from pydantic_ai.exceptions import (
     CostCalculationFailedWarning,
     CostNotFoundWarning,
@@ -27,6 +28,19 @@ from pydantic_ai.exceptions import (
 
 from loom.ai.errors import AgentRunError, AgentRunErrorCode
 
+_UNMEASURED_SPEND_WARNINGS: tuple[type[Warning], ...] = tuple(
+    warning
+    for warning in (
+        CostNotFoundWarning,
+        CostCalculationFailedWarning,
+        getattr(pydantic_ai_exceptions, "UsageExtractionFailedWarning", None),
+    )
+    if warning is not None
+)
+"""pydantic-ai's bare ``Warning`` subclasses, all of them a spend the run could
+not measure. ``UsageExtractionFailedWarning`` first ships in a pydantic-ai
+release newer than the supported floor, so it is mapped only where it exists."""
+
 _STATUS_CODES: Mapping[int, AgentRunErrorCode] = MappingProxyType(
     {
         401: AgentRunErrorCode.UNAUTHORIZED,
@@ -49,13 +63,12 @@ _EXCEPTION_CODES: Mapping[type[Exception], AgentRunErrorCode] = MappingProxyType
         UnexpectedModelBehavior: AgentRunErrorCode.OUTPUT_SCHEMA_VIOLATION,
         ModelAPIError: AgentRunErrorCode.PROVIDER_UNAVAILABLE,
         TimeoutError: AgentRunErrorCode.RUN_TIMEOUT,
-        # Both subclass ``Warning``, not ``UserWarning``; see "Spend caps" in
+        # They subclass ``Warning``, not ``UserWarning``; see "Spend caps" in
         # docs/ai/artifacts.md for why that makes this mapping load-bearing
         # under ``-W error``. ``test_pydantic_ai_errors.py`` walks every
         # ``pydantic_ai._warnings`` subclass of ``Warning`` that is not a
-        # ``UserWarning`` and fails the day a third one is added here unmapped.
-        CostNotFoundWarning: AgentRunErrorCode.COST_NOT_MEASURABLE,
-        CostCalculationFailedWarning: AgentRunErrorCode.COST_NOT_MEASURABLE,
+        # ``UserWarning`` and fails the day a new one is left unmapped.
+        **dict.fromkeys(_UNMEASURED_SPEND_WARNINGS, AgentRunErrorCode.COST_NOT_MEASURABLE),
     }
 )
 

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/ai/engines/pydantic_ai/_limits.py

@@ -15,20 +15,24 @@ model's cost can be permanently unpriceable.
 from __future__ import annotations
 
 import logging
-from dataclasses import dataclass, replace
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from dataclasses import replace
+from decimal import Decimal
 from typing import Any
 
 from genai_prices import calc_price
 from pydantic_ai import RunContext
-from pydantic_ai.capabilities import AbstractCapability
-from pydantic_ai.messages import ModelResponse
-from pydantic_ai.models import ModelRequestContext
-from pydantic_ai.usage import RunUsage, UsageLimits
+from pydantic_ai.messages import ModelMessage, ModelResponse
+from pydantic_ai.models import Model, ModelRequestParameters, StreamedResponse
+from pydantic_ai.models.wrapper import WrapperModel
+from pydantic_ai.settings import ModelSettings
+from pydantic_ai.usage import RequestUsage, RunUsage, UsageLimits
 
 from loom.ai.declarative import PolicySpec
 from loom.ai.pricing import ModelPrice
 
-__all__ = ["ConfiguredPrice", "is_priceable", "usage_limits", "warn_if_model_not_priceable"]
+__all__ = ["PricedModel", "is_priceable", "usage_limits", "warn_if_model_not_priceable"]
 
 _logger = logging.getLogger(__name__)
 
@@ -59,36 +63,58 @@ def usage_limits(policies: PolicySpec) -> UsageLimits:
     )
 
 
-@dataclass
-class ConfiguredPrice(AbstractCapability[Any]):
-    """Price every model response with the deployment's own rates.
+class PricedModel(WrapperModel):
+    """Model whose every response is priced with the deployment's own rates.
 
-    The cost is set before the response joins the run's usage, so the
-    engine's ``cost_limit`` and loom's ``on_unpriced_spend`` both read it,
-    and it replaces whatever genai-prices would have computed.
+    The cost is set on the response the wrapped model returns, before the
+    engine records it in the run's usage, so the engine's ``cost_limit`` and
+    loom's ``on_unpriced_spend`` both read it, whatever the engine version
+    does after the response leaves the model. It replaces any cost the model
+    reported and whatever genai-prices would have computed.
 
     Args:
-        price: Rates of the model the plan is bound to.
+        wrapped: Model built for the plan's binding.
+        price: Rates of that model.
     """
 
-    price: ModelPrice
+    def __init__(self, wrapped: Model, price: ModelPrice) -> None:
+        super().__init__(wrapped)
+        self.price = price
 
-    async def after_model_request(
+    async def request(
         self,
-        ctx: RunContext[Any],
-        *,
-        request_context: ModelRequestContext,
-        response: ModelResponse,
+        messages: list[ModelMessage],
+        model_settings: ModelSettings | None,
+        model_request_parameters: ModelRequestParameters,
     ) -> ModelResponse:
-        """Return *response* with its usage priced at :attr:`price`."""
-        usage = response.usage
-        cost = self.price.cost(
+        """Return the wrapped model's response with its usage priced at :attr:`price`."""
+        response = await super().request(messages, model_settings, model_request_parameters)
+        return replace(response, usage=replace(response.usage, cost=self._cost(response.usage)))
+
+    @asynccontextmanager
+    async def request_stream(
+        self,
+        messages: list[ModelMessage],
+        model_settings: ModelSettings | None,
+        model_request_parameters: ModelRequestParameters,
+        run_context: RunContext[Any] | None = None,
+    ) -> AsyncIterator[StreamedResponse]:
+        """Yield the wrapped model's stream, its usage priced at :attr:`price` once it closes."""
+        async with super().request_stream(
+            messages, model_settings, model_request_parameters, run_context
+        ) as stream:
+            try:
+                yield stream
+            finally:
+                stream.usage.cost = self._cost(stream.usage)
+
+    def _cost(self, usage: RequestUsage) -> Decimal:
+        return self.price.cost(
             input_tokens=usage.input_tokens,
             output_tokens=usage.output_tokens,
             cache_read_tokens=usage.cache_read_tokens,
             cache_write_tokens=usage.cache_write_tokens,
         )
-        return replace(response, usage=replace(usage, cost=cost))
 
 
 def is_priceable(model_name: str, provider_name: str | None) -> bool:

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/ai/engines/pydantic_ai/provider.py

@@ -19,7 +19,6 @@ from __future__ import annotations
 from typing import TYPE_CHECKING, Any, ClassVar
 
 from pydantic_ai import Agent, ModelRetry
-from pydantic_ai.capabilities import AbstractCapability
 from pydantic_ai.models import Model
 
 from loom.ai.abc import AgentEngine, DepsFactory, OutputCheck
@@ -36,7 +35,7 @@ from loom.ai.engines.pydantic_ai._instructions import (
     ensure_templating_available,
 )
 from loom.ai.engines.pydantic_ai._limits import (
-    ConfiguredPrice,
+    PricedModel,
     is_priceable,
     usage_limits,
     warn_if_model_not_priceable,
@@ -164,13 +163,9 @@ class PydanticAIEngineProvider:
         if not isinstance(plan, AgentPlan):
             raise TypeError(f"expected an AgentPlan, got {type(plan).__name__}")
         ensure_templating_available(plan)
-        model = self._resolve_model(plan.inference)
-        if plan.price is None:
-            warn_if_model_not_priceable(
-                model.model_name, _provider_name(model), plan.policies, f"agent '{plan.name}'"
-            )
+        model = _with_price(self._resolve_model(plan.inference), plan)
         toolsets = build_toolsets(plan, container, mcp=self._mcp)
-        capabilities = _with_price(build_capabilities(plan, container), plan)
+        capabilities = build_capabilities(plan, container)
         if not plan.inference.streaming and any(
             capability.has_wrap_run_event_stream for capability in capabilities
         ):
@@ -270,13 +265,14 @@ def _provider_name(model: Model) -> str | None:
     return model.provider.name if model.provider is not None else None
 
 
-def _with_price(
-    capabilities: tuple[AbstractCapability[Any], ...], plan: AgentPlan
-) -> tuple[AbstractCapability[Any], ...]:
-    """Append the plan's configured price, when it declares one, to *capabilities*."""
-    if plan.price is None:
-        return capabilities
-    return (*capabilities, ConfiguredPrice(price=plan.price))
+def _with_price(model: Model, plan: AgentPlan) -> Model:
+    """Price *model* with the plan's configured rates, or warn when it may stay unpriced."""
+    if plan.price is not None:
+        return PricedModel(model, plan.price)
+    warn_if_model_not_priceable(
+        model.model_name, _provider_name(model), plan.policies, f"agent '{plan.name}'"
+    )
+    return model
 
 
 def _register_output_check(agent: Agent[Any, Any], loom_type: LoomType, check: OutputCheck) -> None:

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/__init__.py

@@ -55,6 +55,7 @@ from loom.etl.declarative import (
     CsvReadOptions,
     CsvWriteOptions,
     DeletePolicy,
+    EmptyReplaceError,
     ExcelReadOptions,
     Format,
     FromConfig,
@@ -203,6 +204,7 @@ __all__ = [
     "HistorifyKeyConflictError",
     "HistorifyDateCollisionError",
     "HistorifyTemporalConflictError",
+    "EmptyReplaceError",
     # intermediates
     "CheckpointCleaner",
     "CheckpointScope",

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/backends/_write_policy.py

@@ -15,6 +15,7 @@ from loom.core.logger import get_logger
 from loom.etl.declarative.target import (
     AppendSpec,
     ClientSpec,
+    EmptyReplaceError,
     FileSpec,
     ReplacePartitionsSpec,
     ReplaceSpec,
@@ -188,6 +189,7 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
         streaming: bool,
         create_partition_cols: tuple[str, ...] = (),
         require_physical_partitions: tuple[str, ...] | None = None,
+        allow_empty: bool = True,
     ) -> _PreparedWrite[WriteFrameT, PhysicalSchemaT] | None:
         """Run the shared policy up to the point where the write mode differs.
 
@@ -207,6 +209,10 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
             require_physical_partitions: Partition columns the existing table
                                          must physically carry, or ``None`` to
                                          skip that check.
+            allow_empty:                 When ``False``, raise
+                                         :class:`EmptyReplaceError` for a
+                                         frame with no rows, before the table
+                                         is created or written.
 
         Returns:
             The prepared write when the table already exists and the caller
@@ -220,7 +226,9 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
                 schema_mode=schema_mode,
                 missing_table_policy=self._missing_table_policy,
             )
-            materialized = self._materialize_checked(frame, target, streaming)
+            materialized = self._materialize_checked(
+                frame, target, streaming, allow_empty=allow_empty
+            )
             self._create(
                 materialized,
                 target,
@@ -232,7 +240,7 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
             self._require_physical_partitions(target, require_physical_partitions)
         aligned = self._align(frame, existing, schema_mode)
         return _PreparedWrite(
-            frame=self._materialize_checked(aligned, target, streaming),
+            frame=self._materialize_checked(aligned, target, streaming, allow_empty=allow_empty),
             existing_schema=existing,
         )
 
@@ -260,7 +268,11 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
     ) -> None:
         """Replace policy: check exists → create OR align → write."""
         prepared = self._prepare_write(
-            frame, target, schema_mode=spec.schema_mode, streaming=streaming
+            frame,
+            target,
+            schema_mode=spec.schema_mode,
+            streaming=streaming,
+            allow_empty=spec.allow_empty,
         )
         if prepared is None:
             return
@@ -507,15 +519,29 @@ class _WritePolicy(TargetWriter, Generic[InputFrameT, WriteFrameT, PhysicalSchem
         _ = frame
         return None
 
+    @abstractmethod
+    def _is_empty(self, frame: WriteFrameT) -> bool:
+        """Whether the materialised *frame* has no rows."""
+
     def _materialize_checked(
         self,
         frame: InputFrameT,
         target: ResolvedTarget,
         streaming: bool,
+        *,
+        allow_empty: bool = True,
     ) -> WriteFrameT:
-        """Materialise *frame*, warning when the write carries no rows."""
+        """Materialise *frame*, warning when the write carries no rows.
+
+        Raises:
+            EmptyReplaceError: When *allow_empty* is ``False`` and the
+                materialised frame has no rows.
+        """
         materialized = self._materialize_for_write(frame, streaming)
-        if self._row_count_if_cheap(materialized) == 0:
+        if not allow_empty:
+            if self._is_empty(materialized):
+                raise EmptyReplaceError(target.logical_ref.ref)
+        elif self._row_count_if_cheap(materialized) == 0:
             _log.warning("write_produced_no_rows", target=target.logical_ref.ref)
         return materialized
 

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/backends/polars/__init__.py

@@ -1,7 +1,7 @@
 """Polars backend for the Loom ETL framework."""
 
 from loom.etl.backends.polars._file_writer import PolarsFileWriter
-from loom.etl.backends.polars._reader import PolarsSourceReader
+from loom.etl.backends.polars._reader import PolarsSourceReader, decode_json_columns
 from loom.etl.backends.polars._schema import PolarsPhysicalSchema, read_delta_physical_schema
 from loom.etl.backends.polars._writer import PolarsTargetWriter
 
@@ -10,5 +10,6 @@ __all__ = [
     "PolarsPhysicalSchema",
     "PolarsSourceReader",
     "PolarsTargetWriter",
+    "decode_json_columns",
     "read_delta_physical_schema",
 ]

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/backends/polars/_reader.py

@@ -5,8 +5,8 @@ from __future__ import annotations
 import functools
 import json
 import logging
-from collections.abc import Callable
-from typing import TYPE_CHECKING, Any
+from collections.abc import Callable, Iterable
+from typing import TYPE_CHECKING, Any, TypeVar
 
 if TYPE_CHECKING:
     from loom.etl.io.sources._clickhouse import ClickHouseSourceReader
@@ -23,6 +23,7 @@ from loom.etl.declarative.source import FileSourceSpec, SourceSpec, TableSourceS
 from loom.etl.declarative.source._specs import (
     ClickHouseSourceSpec,
     DynamoDbSourceSpec,
+    JsonColumnSpec,
     MongoSourceSpec,
 )
 from loom.etl.runtime.contracts import SourceReader
@@ -43,6 +44,8 @@ from ._predicate import predicate_to_polars
 
 _log = logging.getLogger(__name__)
 
+_FrameT = TypeVar("_FrameT", pl.DataFrame, pl.LazyFrame)
+
 
 # ---------------------------------------------------------------------------
 # Polymorphic JSON normalization
@@ -124,6 +127,49 @@ def _normalize_json_string(json_str: str | None, *, loom_type: LoomType) -> str
     return json.dumps(result, ensure_ascii=False)
 
 
+def decode_json_columns(frame: _FrameT, json_columns: Iterable[JsonColumnSpec]) -> _FrameT:
+    """Decode JSON string columns the way :class:`PolarsSourceReader` does at read time.
+
+    Each column named in *json_columns* holds JSON text and is replaced by the
+    value decoded into its declared :class:`~loom.etl.schema.LoomType`. Rows
+    are first normalised to that type, so a field that is a string in some rows
+    and an object in others decodes instead of raising
+    :exc:`polars.exceptions.ComputeError`. A null or unparseable value decodes
+    to null. Other columns pass through unchanged.
+
+    Use it to reproduce in a test what a step receives from a source declared
+    with ``parse_json``.
+
+    Args:
+        frame: Eager or lazy frame holding the JSON text columns.
+        json_columns: Column declarations, e.g. ``FromTable(...).json_columns``
+            or ``FromFile(...).json_columns``.
+
+    Returns:
+        A frame of the same kind as *frame*, or *frame* itself when
+        *json_columns* is empty.
+
+    Example::
+
+        from loom.etl.backends.polars import decode_json_columns
+
+        events = FromTable("raw.events").parse_json("payload", Payload)
+        decoded = decode_json_columns(raw_frame, events.json_columns)
+    """
+    exprs = [
+        pl.col(jc.column)
+        .map_elements(
+            functools.partial(_normalize_json_string, loom_type=jc.loom_type),
+            return_dtype=pl.String,
+        )
+        .str.json_decode(loom_type_to_polars(jc.loom_type))
+        for jc in json_columns
+    ]
+    if not exprs:
+        return frame
+    return frame.with_columns(exprs)
+
+
 class PolarsSourceReader(SourceReader):
     """Read table, file, and MongoDB sources using Polars."""
 
@@ -351,21 +397,10 @@ class PolarsSourceReader(SourceReader):
         Applies a per-row normalization pass before ``json_decode`` so that
         polymorphic fields (e.g. ``String`` in some rows, ``Object`` in others)
         are coerced to the declared :class:`~loom.etl.schema.LoomType` without
-        raising :exc:`polars.exceptions.ComputeError`.
+        raising :exc:`polars.exceptions.ComputeError`. See
+        :func:`decode_json_columns`.
         """
-        if not json_columns:
-            return frame
-
-        exprs = [
-            pl.col(jc.column)
-            .map_elements(
-                functools.partial(_normalize_json_string, loom_type=jc.loom_type),
-                return_dtype=pl.String,
-            )
-            .str.json_decode(loom_type_to_polars(jc.loom_type))
-            for jc in json_columns
-        ]
-        return frame.with_columns(exprs)
+        return decode_json_columns(frame, json_columns)
 
 
 def _apply_source_schema(frame: pl.LazyFrame, schema: tuple[ColumnSchema, ...]) -> pl.LazyFrame:
@@ -374,8 +409,8 @@ def _apply_source_schema(frame: pl.LazyFrame, schema: tuple[ColumnSchema, ...])
 
 
 def _apply_json_decode(frame: pl.LazyFrame, json_columns: tuple[Any, ...]) -> pl.LazyFrame:
-    """Compatibility helper used by tests."""
-    return PolarsSourceReader._apply_json_decode(frame, json_columns)
+    """Compatibility alias of :func:`decode_json_columns`."""
+    return decode_json_columns(frame, json_columns)
 
 
 def execute_sql(frames: dict[str, pl.LazyFrame], query: str) -> pl.LazyFrame:
@@ -386,4 +421,10 @@ def execute_sql(frames: dict[str, pl.LazyFrame], query: str) -> pl.LazyFrame:
     return ctx.execute(query, eager=False)
 
 
-__all__ = ["PolarsSourceReader", "execute_sql", "_apply_source_schema", "_apply_json_decode"]
+__all__ = [
+    "PolarsSourceReader",
+    "decode_json_columns",
+    "execute_sql",
+    "_apply_source_schema",
+    "_apply_json_decode",
+]

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/backends/polars/_writer.py

@@ -217,6 +217,10 @@ class PolarsTargetWriter(_WritePolicy[pl.LazyFrame, pl.DataFrame, PolarsPhysical
         """Rows in the collected frame."""
         return frame.height
 
+    def _is_empty(self, frame: pl.DataFrame) -> bool:
+        """Whether the collected frame has no rows."""
+        return frame.is_empty()
+
     def _materialize_for_write(self, frame: pl.LazyFrame, streaming: bool) -> pl.DataFrame:
         """Collect lazy frame to DataFrame before Delta write."""
         df = frame.collect(engine="streaming" if streaming else "auto")

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/backends/spark/_writer.py

@@ -143,6 +143,10 @@ class SparkTargetWriter(_WritePolicy[DataFrame, DataFrame, SparkPhysicalSchema])
         _ = streaming
         return frame
 
+    def _is_empty(self, frame: DataFrame) -> bool:
+        """Whether *frame* has no rows; runs one Spark job that stops at the first row."""
+        return frame.isEmpty()
+
     def _predicate_to_sql(self, predicate: Any, params: Any) -> str:
         """Convert predicate to SQL."""
         return predicate_to_sql(predicate, params)

{loom_kernel-2.15.0 → loom_kernel-2.16.0}/src/loom/etl/compiler/__init__.py

@@ -2,6 +2,7 @@
 
 from loom.etl.compiler._compiler import ETLCompiler
 from loom.etl.compiler._errors import ETLCompilationError, ETLErrorCode
+from loom.etl.compiler._filter import InvalidStageError, filter_plan
 from loom.etl.compiler._plan import (
     AgentBinding,
     ConfigValueBinding,
@@ -13,6 +14,9 @@ from loom.etl.compiler._plan import (
     StepPlan,
     TargetBinding,
     flatten_step_names,
+    iter_all_steps,
+    iter_processes,
+    iter_steps_in_process,
 )
 from loom.etl.compiler._validators import (
     StepCompilationContext,
@@ -31,6 +35,7 @@ __all__ = [
     "ETLCompiler",
     "ETLCompilationError",
     "ETLErrorCode",
+    "InvalidStageError",
     "StepPlan",
     "ProcessPlan",
     "PipelinePlan",
@@ -41,7 +46,11 @@ __all__ = [
     "ParallelStepGroup",
     "ParallelProcessGroup",
     "StepCompilationContext",
+    "filter_plan",
     "flatten_step_names",
+    "iter_all_steps",
+    "iter_processes",
+    "iter_steps_in_process",
     "validate_execute_signature",
     "validate_param_exprs",
     "validate_params_compat",