loom-kernel 2.6.0__tar.gz → 2.8.0__tar.gz

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/CHANGELOG.md

@@ -29,6 +29,10 @@
   and `McpHandle.call(expect=...)` accept a strict `pydantic.BaseModel` next
   to a `msgspec.Struct`; an annotation mixing both libraries is rejected.
 
+- **core:** `section(cfg, key, T)` and `ConfigContext.section` accept a key
+  naming a scalar leaf (`"api.token"`) and convert that value; before, they
+  failed on anything but a mapping.
+
 ### ai
 
 - **ai:** `provider: typesafe` binds TypeSafe's Jev, a decision model that
@@ -88,6 +92,58 @@
   boundaries already use; `LoomRestAdapter`'s Struct projection goes
   through it too.
 
+### prefect
+
+- **prefect:** a deployment may run each time another one completes, inheriting
+  some of its parameters. The per-flow YAML takes an optional `trigger` block:
+  `after`, the upstream deployment as `<flow>/<deployment>` or, for a loom ETL
+  whose flow and deployment share its name, that name; and `inherit_params`, a
+  list of parameter names. The deployer registers it as a Prefect
+  `DeploymentEventTrigger` on `prefect.flow-run.Completed` whose related
+  resources must include both that flow and that deployment, since a deployment
+  name is only unique within its flow. Each inherited parameter is rendered, as
+  text, from the completed run's stored parameters, which loom records resolved,
+  so `${today-1d}` upstream reaches the downstream run as the concrete date.
+  Only fields declared exactly `str`, `date` or `datetime` can be inherited (a
+  null would render as `"None"`), and each must be stored on the upstream run,
+  that is declared under its YAML `params` or submitted: a value it only took
+  from a signature default renders as empty text, which a `str` field takes as
+  `""` and a `date` field fails to decode. Every other parameter takes the
+  deployment's default, and a placeholder among them resolves against the
+  downstream run's creation time. The block is decoded strictly (an unknown key,
+  a wrong type or an `after` with spaces around either part is refused) when the
+  flow is built and, for YAML declarations, as a `ConfigError` before any
+  deployment is registered. A trigger coexists with `schedule`, in which case
+  the ETL runs on both. If the upstream run failed to record its parameters, the
+  downstream run inherits the placeholder text and resolves it against its own
+  creation time. Without `trigger` the deployment is registered exactly as
+  before.
+
+### etl
+
+- **etl:** a step may receive values from the runner's config without reading
+  the environment or taking them as params. `FromConfig("respondio.api_token")`
+  (a `str`) or `FromConfig("respondio", RespondioSettings)` (a type
+  `msgspec.convert` accepts; a null or absent key is reported as not set)
+  declared as a class attribute is passed to
+  `execute()` as the keyword argument of the same name, next to the frames, on
+  `ETLStep` and `ClientStep`. The key is a path into the YAML
+  `ETLRunner.from_yaml` loads, so `${oc.env:...}` and resolvers such as
+  `${secrets:...}` apply. The value is resolved when the step runs:
+  `${oc.env:...}` on every resolution, a resolver's result once per runner,
+  since the config caches it. `ETLCompiler()` checks the shape, with new error
+  codes: `MISSING_CONFIG_PARAMS` (no keyword-only parameter for an attribute),
+  `CONFIG_ALIAS_CONFLICT` (a name shared with a source or with `client`) and
+  `UNSUPPORTED_CONFIG_VALUE` (declared on a `StepSQL`).
+  `ETLCompiler(config_context=...)`, which `ETLRunner.run` uses, checks every
+  key before any step runs (`UNRESOLVED_CONFIG_VALUE`). A plan carries the key
+  and the type only, and loom writes the value to no log, event or checkpoint.
+  A resolution failure names the key and the type, and `ConfigValueError` and
+  `ETLCompilationError` carry no chained exception, since a `msgspec` message
+  may quote the value. `ETLRunner`, `ETLRunner.from_config` and `ETLExecutor` take
+  `config_context=`; `PolarsStepRunner` and `SparkStepRunner` take
+  `with_config(mapping)` for fake values in tests.
+
 ## ⚠ Behaviour changes
 
 ### core
@@ -133,6 +189,44 @@
   OpenAPI schemas for request bodies and responses alike, matching the
   untitled document a Struct always produced.
 
+### prefect
+
+- **prefect:** a placeholder counts from the run's scheduled start instead of
+  the wall clock. `${today-1d}`, `${yesterday}` and `${now±N}` counted from
+  `datetime.now()` inside the flow, so a queued run that started after
+  midnight, or a retry the next day, processed another day than its slot's.
+  `etl_flow`, `maintenance_flow` and `backfill_flow` (`start_from` included)
+  and the run name now resolve against the run's scheduled start
+  (`prefect.runtime.flow_run.scheduled_start_time`, read in UTC): the slot of
+  a scheduled run, the same slot on every retry, the creation time of a manual
+  run. A late run's `${now}` therefore covers up to its scheduled time, not up
+  to the moment it started. Outside a flow run the wall clock still applies.
+  The backfill finalize pass still pins `window_end` to the current chunk.
+- **prefect:** resuming from the manifest now works for placeholder
+  parameters. The correlation id was computed from the resolved value, so
+  with `${now}` it carried the seconds of the clock and a retry never found
+  the manifest of the attempt before it; anchored, every attempt of a run
+  computes the same id. The id is now also computed from the decoded
+  parameters rather than the submitted ones, so a value and its recorded ISO
+  string key the same manifest. Two ids change: a `date` field submitted as
+  an ISO string (`"2026-09-21"` now keys `20260921T000000`, as the `date`
+  itself always did), and a `correlation_field` typed as a plain `Enum`,
+  which now keys the member's sanitised name (`Color_RED`) instead of its
+  submitted value (`red`); a `StrEnum` renders as its value and keeps its id.
+  Do not upgrade across a run you intend to resume from its manifest.
+- **prefect:** the run records the parameters it resolved. Prefect stored and
+  showed the placeholder text, so nothing told which days a run processed.
+  After decoding, the flow updates its own run: the parameters the API stored
+  for it, with each placeholder replaced by its value (dates and datetimes in
+  ISO 8601, a backfill's `start_from` included); nothing is added. The UI
+  shows the concrete window, a run resubmitted from the UI ("Retry") or
+  copied reproduces it, and an automation reading `flow_run.parameters` gets
+  dates. A run that stores no placeholder makes no call, so a resubmitted run
+  makes none. A retry inside the same process (`retries`) still holds the
+  placeholders it started with and repeats the call with the same values,
+  which is harmless. The update is one attempt with a 10-second timeout; if
+  it fails, the flow logs it at `ERROR` in the run's logs and goes on.
+
 ## 🐛 Fixes
 
 ### core

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 2.6.0
+Version: 2.8.0
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Typing :: Typed

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/docs/etl/pipelines.md

@@ -196,6 +196,80 @@ from loom.etl import ETLRunner
 runner = ETLRunner.from_yaml("config/etl.yaml")
 ```
 
+## Config values and secrets (`FromConfig`)
+
+A step that needs a value from the config (an API token, a page size, a whole
+settings block) declares it with `FromConfig`, next to its sources. The
+executor passes the value to `execute()` as a keyword argument named like the
+attribute. The step never reads an environment variable, and the value never
+travels as a pipeline param, so an orchestrator such as Prefect neither shows
+nor stores it.
+
+```python
+import msgspec
+import polars as pl
+
+from loom.etl import ETLParams, ETLStep, FromConfig, IntoTemp
+
+
+class RespondioSettings(msgspec.Struct, kw_only=True):
+    api_token: str
+    page_size: int = 100
+
+
+class FetchMessages(ETLStep[DailyParams]):
+    api_token = FromConfig("respondio.api_token")          # str by default
+    respondio = FromConfig("respondio", RespondioSettings)  # a typed section
+    target = IntoTemp("messages")
+
+    def execute(
+        self,
+        params: DailyParams,
+        *,
+        api_token: str,
+        respondio: RespondioSettings,
+    ) -> pl.LazyFrame: ...
+```
+
+The key is a dot-separated path into the same YAML `ETLRunner.from_yaml` loads,
+outside `storage:`. Interpolations and resolvers apply, so the file decides
+where the value comes from:
+
+```yaml
+storage:
+  engine: polars
+  # ...
+
+respondio:
+  api_token: ${oc.env:RESPONDIO_API_TOKEN}     # injected by the task definition
+  # api_token: ${secrets:/prod/respondio/token} # or AWS Secrets Manager
+  page_size: 200
+```
+
+- **When it resolves.** `ETLRunner.run` compiles the pipeline and checks every
+  `FromConfig` key before any step runs: a missing key, an interpolation that
+  cannot resolve or a value that does not validate as the declared type fails
+  with `ETLCompilationError` (`UNRESOLVED_CONFIG_VALUE`). Each step then
+  resolves its values again when it executes, never at import.
+  `${oc.env:...}` is read on every resolution; a resolver's result
+  (`${secrets:...}`, `${ssm:...}`) is cached by the config for the lifetime of
+  the runner, so a rotated secret takes effect with the next runner.
+- **What is checked without a config.** `ETLCompiler()` alone checks the shape:
+  every `FromConfig` attribute needs a keyword-only parameter of the same name
+  in `execute()`, and may not share its name with a source alias (or with
+  `client` on a `ClientStep`). A `StepSQL` cannot declare one.
+- **Containment.** A plan carries the key and the type, never the value, and
+  loom writes the value to no log, lifecycle event or checkpoint. A resolution
+  failure names the key and the expected type only: `ConfigValueError` (run
+  time) and `ETLCompilationError` (compile time) carry no chained exception,
+  since a `msgspec` message may quote the value. What the step does with the
+  value it receives is up to the step.
+- **Types.** What `msgspec.convert` accepts: `str`, `int`, `Literal[...]`, a
+  `msgspec.Struct` or a dataclass. A null or absent key is reported as not
+  set, so a `FromConfig` value is always required.
+- **Without `from_yaml`.** Pass `config_context=ConfigContext(...)` to
+  `ETLRunner`, `ETLRunner.from_config` or `ETLExecutor`.
+
 ## Write modes
 
 Every `IntoTable` target declares exactly one write mode by chaining a method.

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/docs/etl/testing.md

@@ -23,6 +23,23 @@ def test_clean_orders():
 `StepResult` methods: `assert_schema`, `assert_count`, `assert_not_empty`,
 `show`, `to_polars`.
 
+### Steps that declare `FromConfig` values
+
+`with_config` gives the runner the config a step's `FromConfig` values resolve
+from, so a test passes a fake token instead of setting an environment variable.
+The keys are checked when the step compiles, as `ETLRunner` does.
+`SparkStepRunner` has the same method.
+
+```python
+def test_fetch_uses_the_token():
+    runner = PolarsStepRunner().with_config({"respondio": {"api_token": "fake"}})
+    result = runner.run(FetchMessages, DailyParams(run_date=date(2026, 9, 25)))
+    result.assert_not_empty()
+```
+
+A step whose target the harness cannot capture (an `IntoTemp`, say) can be
+called directly: `FetchMessages().execute(params, api_token="fake", ...)`.
+
 ## Reusable seed datasets with `ETLScenario`
 
 `ETLScenario` stores seed data as plain tuples so the same dataset can be used

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/docs/guides/etl.md

@@ -893,7 +893,7 @@ that matches nothing is a `ConfigError`.
 
 A declaration file has one of two shapes. The first is a **per-ETL document**:
 the YAML `etl_flow` already reads (`schedule`, `params`, `tags`, `environments`,
-`retry`, `notifications`, `correlation_field`) plus the keys that replace the
+`retry`, `notifications`, `correlation_field`, `trigger`) plus the keys that replace the
 Python module:
 
 ```yaml
@@ -958,6 +958,78 @@ declares neither `etls:` nor `pipeline` is a `ConfigError` naming the file, so
 keep ETL declarations in their own directory (`config/etls/`) or in one `etls:`
 file rather than next to the storage YAML.
 
+### Placeholders count from the run's scheduled start
+
+A parameter may hold a placeholder: `${today}`, `${today±Nd}`, `${yesterday}`,
+`${now}` or `${now±N}` with `d`, `h` or `m`. Inside a flow run it counts from
+the run's scheduled start, read in UTC, not from the moment the run executes:
+
+- a scheduled run resolves against its slot, even when it starts late or
+  after midnight;
+- a retry, by the engine or through "Retry" in the UI, resolves against the
+  same slot;
+- a manual run resolves against the time it was created.
+
+A late run's `${now}` therefore covers up to its scheduled time. The run name
+follows the same rule.
+
+Once resolved, the flow writes the values back to its own run: the
+parameters the API stored for it, with each placeholder replaced by its value
+(dates and datetimes in ISO 8601, a backfill's `start_from` included). Nothing
+else is added or changed. The run then shows the window it processed, and a
+run resubmitted from the UI ("Retry") or copied reproduces it exactly: its
+parameters are concrete, so it makes no call. A retry inside the same process
+(`retries`) still holds the placeholders it started with, resolves them to the
+same values and repeats the call, which is harmless.
+
+The update is a single attempt with a 10-second timeout. If it fails, the flow
+logs it at `ERROR`, in the run's logs, and runs anyway; the run then keeps
+showing the placeholders.
+
+Because every attempt of a run resolves the same values, the correlation id,
+and therefore the manifest a retry resumes from, is stable across attempts.
+
+### Running a deployment after another one
+
+A per-flow YAML may name the deployment whose completed runs start this one,
+and the parameters it copies from them:
+
+```yaml
+trigger:
+  after: respondio_messages_daily
+  inherit_params: [updated_at_from, updated_at_to]
+```
+
+`after` is the upstream deployment as `<flow>/<deployment>`, or a single name
+when the flow and the deployment share it, as every loom ETL's do. The
+deployer registers a Prefect `DeploymentEventTrigger` that fires on
+`prefect.flow-run.Completed` of a run related to both that flow and that
+deployment; a deployment name alone is only unique within its flow.
+
+Each name in `inherit_params` is copied from the completed run's stored
+parameters. Since the run records its placeholders resolved, the downstream
+run receives the same concrete window. The name must therefore be stored on the
+upstream run, which means declared under the upstream YAML's `params` (or
+submitted with the run): a value the upstream flow only took from its
+signature default is not stored, so it renders as empty text, which a `str`
+field accepts as `""` and a `date` or `datetime` field fails to decode. The trigger passes each value as text,
+so only a field declared exactly `str`, `date` or `datetime` can be inherited;
+an optional field is refused because a null would arrive as the text `"None"`.
+
+- Parameters not inherited take this deployment's defaults; a placeholder
+  among them resolves against the downstream run's own creation time.
+- If the upstream run could not record its parameters, the downstream run
+  inherits the placeholder text and resolves it against its own creation
+  time, which may not be the upstream slot.
+- `trigger` and `schedule` may both be set, and the ETL then runs on both: on
+  its schedule and after each completed upstream run.
+- The upstream name is not checked against the deployments that exist: a
+  trigger naming no deployment never fires.
+
+The block is decoded strictly: `after` and `inherit_params` are its only keys,
+`after` must name a deployment, without surrounding spaces in either part, and
+each inherited name must be a field of the flow's params type.
+
 ### Validation before any deployment
 
 Every declaration is read and checked before the first deployment is

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/core/config/context.py

@@ -169,8 +169,11 @@ class ConfigContext:
     def section(self, key: str | ConfigKey, target: type[T]) -> T:
         """Extract and validate a typed config section by dot-path.
 
+        A key naming a scalar leaf (``"api.token"``) converts that value.
+
         Args:
-            key: Dot-separated path (e.g. ``"database"`` or ``"services.cache"``).
+            key: Dot-separated path (e.g. ``"database"``, ``"services.cache"``
+                or ``"api.token"``).
             target: Type to convert the section into.
 
         Returns:

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/core/config/loader.py

@@ -568,12 +568,13 @@ def section(cfg: DictConfig, key: str, target_type: type[T]) -> T:
     the result to ``target_type`` via ``msgspec.convert``.
 
     Works with any type supported by ``msgspec.convert``: ``msgspec.Struct``
-    subclasses, ``dataclasses``, ``TypedDict``, plain dicts, etc.
+    subclasses, ``dataclasses``, ``TypedDict``, plain dicts, etc.  A key that
+    names a scalar leaf (``"api.token"``) converts that value, e.g. to ``str``.
 
     Args:
         cfg: Root :class:`omegaconf.DictConfig` returned by :func:`load_config`.
-        key: Dot-separated path to the desired section (e.g. ``"database"``
-            or ``"services.cache"``).
+        key: Dot-separated path to the desired section or scalar leaf (e.g.
+            ``"database"`` or ``"services.cache"``).
         target_type: Type to convert the section into.
 
     Returns:
@@ -602,6 +603,8 @@ def section(cfg: DictConfig, key: str, target_type: type[T]) -> T:
                 f"Config section not found: {key!r}  (failed at segment {part!r})"
             ) from exc
 
+    if not omega_conf.is_config(node):
+        return _convert_section(node, key, target_type)
     try:
         data = omega_conf.to_container(node, resolve=True)
     except Exception as exc:
@@ -610,6 +613,15 @@ def section(cfg: DictConfig, key: str, target_type: type[T]) -> T:
             f"(check ${{oc.env:VAR}} interpolations): {exc}"
         ) from exc
 
+    return _convert_section(data, key, target_type)
+
+
+def _convert_section(data: Any, key: str, target_type: type[T]) -> T:
+    """Convert resolved section data to ``target_type`` leniently.
+
+    Raises:
+        ConfigError: When ``msgspec`` validation fails.
+    """
     try:
         return msgspec.convert(data, target_type, strict=False)
     except msgspec.ValidationError as exc:

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/etl/__init__.py

@@ -14,6 +14,7 @@ Authoring::
         SourceSet,
         FromTable,
         FromFile,
+        FromConfig,
         IntoTable,
         IntoFile,
         Format,
@@ -55,6 +56,7 @@ from loom.etl.declarative import (
     DeletePolicy,
     ExcelReadOptions,
     Format,
+    FromConfig,
     FromFile,
     FromTable,
     FromTemp,
@@ -171,6 +173,7 @@ __all__ = [
     "FromTable",
     "FromFile",
     "FromTemp",
+    "FromConfig",
     "FromClickHouse",
     "FromMongo",
     "FromDynamoDb",

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/etl/compiler/__init__.py

@@ -3,6 +3,7 @@
 from loom.etl.compiler._compiler import ETLCompiler
 from loom.etl.compiler._errors import ETLCompilationError, ETLErrorCode
 from loom.etl.compiler._plan import (
+    ConfigValueBinding,
     ParallelProcessGroup,
     ParallelStepGroup,
     PipelinePlan,
@@ -33,6 +34,7 @@ __all__ = [
     "ProcessPlan",
     "PipelinePlan",
     "SourceBinding",
+    "ConfigValueBinding",
     "TargetBinding",
     "ParallelStepGroup",
     "ParallelProcessGroup",

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/etl/compiler/_binding.py

@@ -9,7 +9,7 @@ from enum import Enum
 from typing import Any, Protocol, runtime_checkable
 
 from loom.etl.compiler._errors import ETLCompilationError
-from loom.etl.compiler._plan import SourceBinding, TargetBinding
+from loom.etl.compiler._plan import ConfigValueBinding, SourceBinding, TargetBinding
 from loom.etl.pipeline._step import ETLStep
 
 
@@ -74,6 +74,22 @@ def resolve_target_binding(step_type: type[ETLStep[Any]]) -> TargetBinding:
     return TargetBinding(spec=target._to_spec())
 
 
+def resolve_config_bindings(step_type: type[ETLStep[Any]]) -> tuple[ConfigValueBinding, ...]:
+    """Build config value bindings from the step's ``FromConfig`` attributes.
+
+    Args:
+        step_type: Concrete ``ETLStep`` subclass.
+
+    Returns:
+        Tuple of :class:`~loom.etl.compiler._plan.ConfigValueBinding`, in
+        declaration order.
+    """
+    return tuple(
+        ConfigValueBinding(alias=alias, key=value.key, value_type=value.value_type)
+        for alias, value in step_type._config_values.items()
+    )
+
+
 # ---------------------------------------------------------------------------
 # Internals
 # ---------------------------------------------------------------------------

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/etl/compiler/_compiler.py

@@ -12,10 +12,12 @@ Static validation
 * ``execute()`` first positional param is typed as ``ParamsT``
 * every source alias has a matching ``*``-only DataFrame param in ``execute()``
 * every ``*``-only param in ``execute()`` has a matching source alias
+* every ``FromConfig`` attribute has a matching ``*``-only param in ``execute()``
 * ``target`` is declared
 * source forms are not mixed (caught earlier in ``__init_subclass__``)
 * ``ETLProcess.steps`` entries are valid step types
 * ``ETLPipeline.processes`` entries are valid process types
+* with a config context, every ``FromConfig`` key resolves and validates
 """
 
 from __future__ import annotations
@@ -23,7 +25,12 @@ from __future__ import annotations
 import logging
 from typing import Any, cast
 
-from loom.etl.compiler._binding import resolve_source_bindings, resolve_target_binding
+from loom.core.config import ConfigContext
+from loom.etl.compiler._binding import (
+    resolve_config_bindings,
+    resolve_source_bindings,
+    resolve_target_binding,
+)
 from loom.etl.compiler._errors import ETLCompilationError
 from loom.etl.compiler._plan import (
     ParallelProcessGroup,
@@ -43,6 +50,11 @@ from loom.etl.compiler._validators import (
     validate_step,
     validate_step_catalog,
 )
+from loom.etl.compiler._validators_config import (
+    validate_plan_config,
+    validate_process_config,
+    validate_step_config,
+)
 from loom.etl.declarative.target import AppendSpec
 from loom.etl.pipeline._pipeline import ETLPipeline
 from loom.etl.pipeline._process import ETLProcess
@@ -63,6 +75,9 @@ class ETLCompiler:
         catalog: Optional :class:`~loom.etl._io.TableDiscovery` instance.
                  When provided, all ``TABLE`` sources and ``IntoTable`` targets
                  are validated against the catalog at compile time.
+        config_context: Optional config the runner was built from.  When
+                 provided, every ``FromConfig`` key is resolved and validated
+                 at compile time; values are discarded, never stored on a plan.
 
     Example::
 
@@ -70,8 +85,13 @@ class ETLCompiler:
         plan = ETLCompiler(catalog=HiveCatalog()).compile(DailyOrdersPipeline)
     """
 
-    def __init__(self, catalog: TableDiscovery | None = None) -> None:
+    def __init__(
+        self,
+        catalog: TableDiscovery | None = None,
+        config_context: ConfigContext | None = None,
+    ) -> None:
         self._catalog = catalog
+        self._config_context = config_context
         self._step_cache: dict[type[Any], StepPlan] = {}
         self._process_cache: dict[type[Any], ProcessPlan] = {}
 
@@ -85,7 +105,8 @@ class ETLCompiler:
             Fully validated :class:`~loom.etl.compiler._plan.PipelinePlan`.
 
         Raises:
-            ETLCompilationError: If any structural constraint is violated.
+            ETLCompilationError: If any structural constraint is violated, or
+                a ``FromConfig`` key does not resolve against the config context.
         """
         params_type = _require_params_type(pipeline_type, "ETLPipeline")
         _log.debug(
@@ -99,6 +120,8 @@ class ETLCompiler:
         validate_plan_temps(plan)
         if self._catalog is not None:
             validate_plan_catalog(plan, self._catalog)
+        if self._config_context is not None:
+            validate_plan_config(plan, self._config_context)
         return plan
 
     def compile_process(self, process_type: type[ETLProcess[Any]]) -> ProcessPlan:
@@ -111,11 +134,14 @@ class ETLCompiler:
             Validated :class:`~loom.etl.compiler._plan.ProcessPlan`.
 
         Raises:
-            ETLCompilationError: If any structural constraint is violated.
+            ETLCompilationError: If any structural constraint is violated, or
+                a ``FromConfig`` key does not resolve against the config context.
         """
         plan = self._get_or_build_process(process_type)
         if self._catalog is not None:
             validate_process_catalog(plan, self._catalog)
+        if self._config_context is not None:
+            validate_process_config(plan, self._config_context)
         return plan
 
     def compile_step(self, step_type: type[ETLStep[Any]]) -> StepPlan:
@@ -128,11 +154,14 @@ class ETLCompiler:
             Validated :class:`~loom.etl.compiler._plan.StepPlan`.
 
         Raises:
-            ETLCompilationError: If any structural constraint is violated.
+            ETLCompilationError: If any structural constraint is violated, or
+                a ``FromConfig`` key does not resolve against the config context.
         """
         plan = self._get_or_build_step(step_type)
         if self._catalog is not None:
             validate_step_catalog(plan, self._catalog)
+        if self._config_context is not None:
+            validate_step_config(plan, self._config_context)
         return plan
 
     # ------------------------------------------------------------------
@@ -203,6 +232,7 @@ class ETLCompiler:
         params_type = _require_params_type(step_type, "ETLStep")
         source_bindings = resolve_source_bindings(step_type)
         target_binding = resolve_target_binding(step_type)
+        config_bindings = resolve_config_bindings(step_type)
         _log.debug(
             "compile step=%s sources=%d",
             step_type.__name__,
@@ -214,6 +244,7 @@ class ETLCompiler:
                 params_type=params_type,
                 source_bindings=source_bindings,
                 target_binding=target_binding,
+                config_bindings=config_bindings,
             )
         )
         _warn_append_target(step_type, target_binding.spec)
@@ -223,6 +254,7 @@ class ETLCompiler:
             source_bindings=source_bindings,
             target_binding=target_binding,
             streaming=step_type.streaming,
+            config_bindings=config_bindings,
         )
 
     # ------------------------------------------------------------------

{loom_kernel-2.6.0 → loom_kernel-2.8.0}/src/loom/etl/compiler/_errors.py

@@ -39,6 +39,10 @@ class ETLErrorCode(StrEnum):
     INVALID_TEMP_APPEND_MIX = "INVALID_TEMP_APPEND_MIX"
     UNKNOWN_PARAM_FIELD = "UNKNOWN_PARAM_FIELD"
     UNKNOWN_TEMPLATE_FIELD = "UNKNOWN_TEMPLATE_FIELD"
+    MISSING_CONFIG_PARAMS = "MISSING_CONFIG_PARAMS"
+    CONFIG_ALIAS_CONFLICT = "CONFIG_ALIAS_CONFLICT"
+    UNSUPPORTED_CONFIG_VALUE = "UNSUPPORTED_CONFIG_VALUE"
+    UNRESOLVED_CONFIG_VALUE = "UNRESOLVED_CONFIG_VALUE"
 
 
 class ETLCompilationError(Exception):
@@ -112,13 +116,13 @@ class ETLCompilationError(Exception):
 
     @classmethod
     def extra_source_params(cls, step: type, extra: frozenset[str]) -> ETLCompilationError:
-        """Kw-only execute() parameters not declared in sources."""
+        """Kw-only execute() parameters not declared in sources nor as FromConfig."""
         return cls(
             code=ETLErrorCode.EXTRA_SOURCE_PARAMS,
             component=step.__qualname__,
             message=(
                 f"{step.__qualname__}.execute: parameter(s) {sorted(extra)} "
-                "declared after '*' but not found in sources"
+                "declared after '*' but not found in sources or FromConfig values"
             ),
         )
 
@@ -349,3 +353,53 @@ class ETLCompilationError(Exception):
             ),
             field=field_name,
         )
+
+    @classmethod
+    def missing_config_params(cls, step: type, missing: frozenset[str]) -> ETLCompilationError:
+        """FromConfig attributes absent as kw-only execute() parameters."""
+        return cls(
+            code=ETLErrorCode.MISSING_CONFIG_PARAMS,
+            component=step.__qualname__,
+            message=(
+                f"{step.__qualname__}.execute: FromConfig value(s) {sorted(missing)} "
+                "declared on the step but missing as keyword-only parameter(s) after '*'"
+            ),
+        )
+
+    @classmethod
+    def config_alias_conflict(cls, step: type, aliases: frozenset[str]) -> ETLCompilationError:
+        """A FromConfig attribute shares its name with a source or the injected client."""
+        return cls(
+            code=ETLErrorCode.CONFIG_ALIAS_CONFLICT,
+            component=step.__qualname__,
+            message=(
+                f"{step.__qualname__}: FromConfig value(s) {sorted(aliases)} share their "
+                "name with a source alias or the injected 'client'; rename the attribute"
+            ),
+        )
+
+    @classmethod
+    def unsupported_config_value(cls, step: type) -> ETLCompilationError:
+        """A StepSQL declares FromConfig values it has no execute() to receive."""
+        return cls(
+            code=ETLErrorCode.UNSUPPORTED_CONFIG_VALUE,
+            component=step.__qualname__,
+            message=(
+                f"{step.__qualname__}: a StepSQL renders SQL and never calls execute(), "
+                "so it cannot receive FromConfig values"
+            ),
+        )
+
+    @classmethod
+    def unresolved_config_value(cls, step: type, alias: str, reason: str) -> ETLCompilationError:
+        """A FromConfig key cannot be supplied from the runner's config.
+
+        *reason* must never carry the value: callers pass the message of a
+        :class:`~loom.etl.runtime.ConfigValueError`.
+        """
+        return cls(
+            code=ETLErrorCode.UNRESOLVED_CONFIG_VALUE,
+            component=step.__qualname__,
+            message=f"{step.__qualname__}.{alias}: {reason}",
+            field=alias,
+        )