loom-kernel 0.11.0__tar.gz → 0.11.2__tar.gz

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/CHANGELOG.md

@@ -1,3 +1,65 @@
+# 🚀 Release 0.11.2 ([#58](https://github.com/the-reacher-data/loom-py/pull/58)) ([`54b6069`](https://github.com/the-reacher-data/loom-py/commit/54b6069e4b012e1ad64e8da3baaf18afe735afc9))
+
+
+
+## 🐛 Fixes
+### historify
+- **historify:** allow recurring tracked values in LOG mode
+
+
+
+
+
+
+
+
+
+
+# 🚀 Release 0.11.1 ([#56](https://github.com/the-reacher-data/loom-py/pull/56)) ([`817a6c4`](https://github.com/the-reacher-data/loom-py/commit/817a6c48c6d377ca21bf60cede1c16126a8fdcfb))
+
+
+
+## 🐛 Fixes
+### config
+- **config:** resolve interpolations in includes paths<br>
+  > `_load_local_file` was calling `omega_conf.to_container(raw_includes)` without
+  > `resolve=True`, so paths like `includes: [pool.${oc.env:VARIANT,local}.yaml]`
+  > were treated literally and raised ConfigError on the next load attempt.
+  > Aligns the call with `loader.py:325`, which already passes `resolve=True`
+  > when materialising the resolved config. The facade default stays
+  > `resolve=False` to preserve other callers that need unresolved data.
+  > Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
+
+
+### polars
+- **polars:** skip create on empty frame missing partition cols<br>
+  > `replace_partitions` already skips writing when the frame is empty on the
+  > update path (existing table). The create path (target does not exist yet)
+  > did not, so a fallback that produced an empty frame without the partition
+  > columns surfaced a cryptic delta-rs `SchemaMismatchError: Unable to get
+  > field named "<col>"`.
+  > Two changes in `PolarsTargetWriter._create`:
+  > 1. If partition cols are missing AND the frame is empty: skip with a
+  > warning. Mirrors `_replace_partitions` / `_streaming_replace_partitions`
+  > so create and update behave consistently.
+  > 2. If partition cols are missing AND the frame has rows: raise a clear
+  > loom `ValueError` instead of letting delta-rs fail with an opaque
+  > schema error. The message lists the missing cols and the frame schema
+  > so the caller can diagnose without inspecting tracebacks.
+  > Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
+  > --------
+  > Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
+
+
+
+
+
+
+
+
+
+
+
 # 🚀 Release 0.11.0 ([#54](https://github.com/the-reacher-data/loom-py/pull/54)) ([`23bff3d`](https://github.com/the-reacher-data/loom-py/commit/23bff3df20607bc12268de0d941ce46df61e16bf))
 
 

loom_kernel-0.11.2/CHANGELOG_RELEASE.md

@@ -0,0 +1,15 @@
+# 🚀 Release 0.11.2 ([#58](https://github.com/the-reacher-data/loom-py/pull/58)) ([`54b6069`](https://github.com/the-reacher-data/loom-py/commit/54b6069e4b012e1ad64e8da3baaf18afe735afc9))
+
+
+
+## 🐛 Fixes
+### historify
+- **historify:** allow recurring tracked values in LOG mode
+
+
+
+
+
+
+
+

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loom-kernel
-Version: 0.11.0
+Version: 0.11.2
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Typing :: Typed

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "loom-kernel"
-version = "0.11.0"
+version = "0.11.2"
 description = "Loom Python project"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -268,7 +268,7 @@ markers = [
 
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.11.0"
+version = "0.11.2"
 tag_format = "v$version"
 version_files = [
     "pyproject.toml:project.version",

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/src/loom/core/config/loader.py

@@ -178,7 +178,7 @@ def _load_local_file(path: str, omega_conf: Any, seen: set[str]) -> Any:
     if raw_includes is None:
         return cfg
 
-    include_paths = list(omega_conf.to_container(raw_includes))
+    include_paths = list(omega_conf.to_container(raw_includes, resolve=True))
     cfg = omega_conf.masked_copy(cfg, [k for k in cfg if k != "includes"])
 
     base_dir = Path(resolved).parent

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/src/loom/etl/backends/_historify/_transform.py

@@ -48,8 +48,9 @@ def scd2_transform(
     join_key = list(spec.keys) + list(track_cols)
     eff_date = resolve_effective_date(spec, params_instance)
 
-    ops.assert_unique_keys(frame, join_key)
-    if spec.mode is HistorifyInputMode.LOG:
+    is_log = spec.mode is HistorifyInputMode.LOG
+    ops.assert_unique_keys(frame, [*spec.keys, str(spec.effective_date)] if is_log else join_key)
+    if is_log:
         ops.assert_no_date_collisions(frame, join_key, str(spec.effective_date), spec)
 
     if existing is None:

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/src/loom/etl/backends/polars/_writer.py

@@ -303,6 +303,25 @@ class PolarsTargetWriter(_WritePolicy[pl.LazyFrame, pl.DataFrame, PolarsPhysical
         path_target = self._as_path_target(target)
         _ = schema_mode
         location = path_target.location
+        if partition_cols:
+            missing = tuple(c for c in partition_cols if c not in frame.columns)
+            if missing:
+                # Empty frame with no partition cols is a legitimate no-op
+                # (e.g. decomposition fallback when the source field is absent
+                # from the input schema). Mirrors _replace_partitions on the
+                # update path.
+                if frame.is_empty():
+                    _log.warning(
+                        "create skipped: empty frame missing partition columns",
+                        table=location.uri,
+                        missing=list(missing),
+                    )
+                    return
+                raise ValueError(
+                    f"_create: partition columns not found in frame schema: "
+                    f"{list(missing)}. Frame columns: {frame.columns}. Cannot "
+                    f"create partitioned Delta table at {location.uri!r}."
+                )
         self._warn_uc_first_create(path_target)
         kwargs = self._write_kwargs(location)
 

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/tests/unit/core/config/test_config.py

@@ -313,6 +313,33 @@ def test_includes_missing_file_raises(tmp_path: Path) -> None:
         load_config(str(main))
 
 
+def test_includes_resolves_env_interpolation_in_path(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    monkeypatch.setenv("VARIANT", "prod")
+    (tmp_path / "values.prod.yaml").write_text("database:\n  url: postgresql://prod/db\n")
+    main = tmp_path / "app.yaml"
+    main.write_text("includes:\n  - values.${oc.env:VARIANT}.yaml\napp:\n  name: myapp\n")
+
+    cfg = load_config(str(main))
+
+    assert cfg.database.url == "postgresql://prod/db"
+    assert cfg.app.name == "myapp"
+
+
+def test_includes_resolves_env_interpolation_default_when_var_unset(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    monkeypatch.delenv("VARIANT", raising=False)
+    (tmp_path / "values.local.yaml").write_text("flag: from_default\n")
+    main = tmp_path / "app.yaml"
+    main.write_text("includes:\n  - values.${oc.env:VARIANT,local}.yaml\n")
+
+    cfg = load_config(str(main))
+
+    assert cfg.flag == "from_default"
+
+
 # ---------------------------------------------------------------------------
 # load_config — cloud URIs
 # ---------------------------------------------------------------------------

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/tests/unit/etl/backends/_historify_contract.py

@@ -28,6 +28,7 @@ from loom.etl.declarative.expr._refs import TableRef
 from loom.etl.declarative.target._history import (
     DeletePolicy,
     HistorifyInputMode,
+    HistorifyKeyConflictError,
     HistorifySpec,
     HistorifyTemporalConflictError,
 )
@@ -810,6 +811,66 @@ class HistorifyContractTests:
         assert len(sub2) == 1
         assert sub2[0]["valid_to"] is None
 
+    def test_log_recurring_value_creates_three_versions(
+        self,
+        writer: Any,
+        root: Path,
+        make_frame: Callable,
+        read_table: Callable,
+    ) -> None:
+        frame = make_frame(
+            [
+                {"subscription_id": 1, "plan": "basic", "event_date": date(2024, 1, 1)},
+                {"subscription_id": 1, "plan": "pro", "event_date": date(2024, 6, 1)},
+                {"subscription_id": 1, "plan": "basic", "event_date": date(2024, 9, 1)},
+            ]
+        )
+        writer.write(frame, _log_spec(), None)
+        rows = sorted(read_table(self._uri(root, "dim_subs")), key=lambda r: r["valid_from"])
+        assert len(rows) == 3
+        assert [r["plan"] for r in rows] == ["basic", "pro", "basic"]
+        assert [r["valid_from"] for r in rows] == [
+            date(2024, 1, 1),
+            date(2024, 6, 1),
+            date(2024, 9, 1),
+        ]
+        assert [r["valid_to"] for r in rows] == [date(2024, 5, 31), date(2024, 8, 31), None]
+
+    def test_log_recurrence_replay_is_idempotent(
+        self,
+        writer: Any,
+        root: Path,
+        make_frame: Callable,
+        read_table: Callable,
+    ) -> None:
+        events = [
+            {"subscription_id": 1, "plan": "basic", "event_date": date(2024, 1, 1)},
+            {"subscription_id": 1, "plan": "pro", "event_date": date(2024, 6, 1)},
+            {"subscription_id": 1, "plan": "basic", "event_date": date(2024, 9, 1)},
+        ]
+        writer.write(make_frame(events), _log_spec(), None)
+        writer.write(make_frame(events), _log_spec(), None)  # replay
+        rows = read_table(self._uri(root, "dim_subs"))
+        assert len(rows) == 3
+        froms = [r["valid_from"] for r in rows]
+        assert len(set(froms)) == 3
+        for r in rows:
+            assert r["valid_to"] is None or r["valid_to"] >= r["valid_from"]
+
+    def test_log_same_instant_duplicate_raises(
+        self,
+        writer: Any,
+        make_frame: Callable,
+    ) -> None:
+        frame = make_frame(
+            [
+                {"subscription_id": 1, "plan": "basic", "event_date": date(2024, 1, 1)},
+                {"subscription_id": 1, "plan": "pro", "event_date": date(2024, 1, 1)},
+            ]
+        )
+        with pytest.raises(HistorifyKeyConflictError):
+            writer.write(frame, _log_spec(), None)
+
     # ------------------------------------------------------------------
     # Overwrite columns
     # ------------------------------------------------------------------

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/tests/unit/etl/backends/test_polars/test_backend.py

@@ -379,6 +379,54 @@ def test_writer_replace_partitions_first_run_creates_partitioned_table(tmp_path:
     assert dt.metadata().partition_columns == ["year"]
 
 
+def test_writer_replace_partitions_first_run_empty_frame_missing_partition_cols_skips(
+    tmp_path: Path,
+) -> None:
+    """Empty frame whose schema omits the partition columns must be a no-op.
+
+    This is the case produced by decomposition fallbacks that emit an empty
+    frame with only the root id column when the source field is absent. The
+    create path used to surface a cryptic delta-rs error ("Partition column(s)
+    not found in write schema"); it should silently skip instead, mirroring
+    ``_replace_partitions`` on the update path.
+    """
+    writer = PolarsTargetWriter(tmp_path)
+    # No "year" column at all — empty schema, empty rows. Mimics _empty_frame().
+    empty = pl.DataFrame({"transaction_id": []}, schema={"transaction_id": pl.Utf8})
+    spec = ReplacePartitionsSpec(
+        table_ref=TableRef("staging.empty_no_partition_cols"),
+        partition_cols=("year",),
+        schema_mode=SchemaMode.OVERWRITE,
+    )
+
+    writer.write(empty.lazy(), spec, None)
+
+    target_dir = table_path(tmp_path, TableRef("staging.empty_no_partition_cols"))
+    assert not target_dir.exists() or not any(target_dir.iterdir())
+
+
+def test_writer_replace_partitions_first_run_missing_partition_cols_raises_loom_error(
+    tmp_path: Path,
+) -> None:
+    """Non-empty frame missing partition columns must raise a clear loom error.
+
+    Without this guard the delta-rs error ``Partition column(s) not found in
+    write schema`` surfaces to callers and gives no hint that the frame is the
+    cause.
+    """
+    writer = PolarsTargetWriter(tmp_path)
+    # Has data, but partition column "year" is absent from the schema.
+    frame = pl.DataFrame({"v": [1, 2]})
+    spec = ReplacePartitionsSpec(
+        table_ref=TableRef("staging.missing_partition"),
+        partition_cols=("year",),
+        schema_mode=SchemaMode.OVERWRITE,
+    )
+
+    with pytest.raises(ValueError, match="partition columns.*not found.*year"):
+        writer.write(frame.lazy(), spec, None)
+
+
 def test_writer_replace_where_overwrites_matching_rows(tmp_path: Path) -> None:
     initial = pl.DataFrame({"year": [2023, 2024], "v": [10, 20]})
     _seed(tmp_path, "staging.yearly", initial)

{loom_kernel-0.11.0 → loom_kernel-0.11.2}/uv.lock

@@ -1583,7 +1583,7 @@ wheels = [
 
 [[package]]
 name = "loom-kernel"
-version = "0.11.0"
+version = "0.11.2"
 source = { editable = "." }
 dependencies = [
     { name = "aiocache" },

loom_kernel-0.11.0/CHANGELOG_RELEASE.md

@@ -1,162 +0,0 @@
-# 🚀 Release 0.11.0 ([#54](https://github.com/the-reacher-data/loom-py/pull/54)) ([`23bff3d`](https://github.com/the-reacher-data/loom-py/commit/23bff3df20607bc12268de0d941ce46df61e16bf))
-
-
-## ✨ Features
-### maintenance
-- **maintenance:** add loom.etl.maintenance — agnostic Delta vacuum/optimize<br>
-  > New module providing a declarative, backend-agnostic API for running
-  > Delta table maintenance operations (vacuum, compact, z-order) via delta-rs.
-  > Core abstractions:
-  > MaintenanceStep[ParamsT] base class (mirrors ETLStep pattern)
-  > MaintainTable fluent builder for explicit per-table ops
-  > MaintainSchema for autodiscovery by schema prefix from StorageConfig
-  > MaintenanceRunner.from_config() wired to StorageConfig.to_path_locator()
-  > MaintenanceReport with per-table error isolation + raise_if_errors()
-  > operations_for(params) hook enabling Prefect-parameter-driven operations
-  > Storage config:
-  > MaintenanceConfig + MaintenanceVacuumConfig added to StorageConfig
-  > run_from_config() for config-only maintenance without any Python class
-  > Prefect integration:
-  > maintenance_flow() factory in loom.prefect.flow (mirrors etl_flow)
-  > _common.coerce_tags extracted from _factory — no more private cross-import
-  > Deployable via same YAML pattern as ETL pipelines (step_class key)
-  > Safety defaults:
-  > vacuum dry_run=True everywhere; production must opt-in with dry_run=False
-  > continue-on-error per table; MaintenanceReport.raise_if_errors() for gates
-  > missing_table_policy="skip" by default
-  > 42 unit tests covering builder, step, runner (StubMaintainer), and
-  > DeltaRsMaintainer backend against real local Delta tables.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-
-### notify
-- **notify:** enrich NotifyEvent with duration + env; fix loggers<br>
-  > NotifyEvent:
-  > duration_seconds: float | None — wall-clock time from Prefect's
-  > flow_run.total_run_time. Displayed as '2m 03s', '1h 05m 12s', etc.
-  > env: str | None — forwarded from the flow's 'env' parameter so ops
-  > teams know which environment triggered the alert.
-  > _hooks.py:
-  > _duration_from_run() extracts total_run_time.total_seconds()
-  > _env_from_params() reads the 'env' param (present in all loom flows)
-  > correlation_id stays empty for maintenance flows (no business key) —
-  > the run name already carries the timestamp identifier
-  > Replace stdlib logging with loom get_logger; convert %s → kwargs
-  > SlackNotifier._render:
-  > Header: icon + flow_name + state
-  > Meta line: run name  env (when set)  correlation (when set)  duration
-  > Omits empty fields — maintenance messages stay clean without correlation
-  > _fmt_duration() helper: 45s / 2m 03s / 1h 05m 12s
-  > dry_run indicator is in the run name for maintenance (Prefect shows
-  > params in the UI); a future iteration can surface it in the message
-  > field via MaintenanceReport summary.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **notify:** add run summary to Slack — ETL steps + maintenance table count
-
-
-## 🐛 Fixes
-### maintenance
-- **maintenance:** use loom StructLogger + fix logging.exception + Sonar smells<br>
-  > _runner.py, backends/_delta_rs.py: replace stdlib `import logging` with
-  > loom's `from loom.core.logger import get_logger`. StructLogger only accepts
-  > (event: str, **kwargs) — convert all %s positional format calls to keyword
-
-
-### test
-- **test:** use type() instead of class statement to avoid Sonar S1118<br>
-  > type() triggers __init_subclass__ identically to a class statement but
-  > returns a value — static analyzers see a function call, not an unused
-  > class declaration. Resolves Sonar 'Remove this unused class declaration'
-  > on test_step.py:28 and :34.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **test:** revert to class statement with NOSONAR — type() introduced Bug Major<br>
-  > Using type() to avoid Sonar S1118 introduced a new Bug Major ('return
-  > value of type() must be used'). The class statement IS the correct pattern
-  > for testing __init_subclass__ — the class definition is the act under test.
-  > Add # NOSONAR on the class line: SonarQube's official suppression mechanism
-  > for confirmed false positives. Tells Sonar the unused class is intentional.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-
-
-
-
-## ♻️ Refactor
-### maintenance
-- **maintenance:** move imports to module level + fix noqa:ARG002<br>
-  > _delta_rs.py: move deltalake + protocol types to module level; this module
-  > is only loaded when DeltaRsMaintainer is used so the [delta] extra is always
-  > present. Removes lazy imports from all three method bodies.
-  > _runner.py: move _ops types (CompactSpec, MaintenanceSpec, VacuumSpec,
-  > ZOrderSpec) and TableRef to module level. Keep DeltaRsMaintainer import
-  > lazy inside from_config() with noqa:PLC0415 — avoids forcing [delta] on
-  > users who import MaintenanceRunner for type annotations only.
-  > Remove incorrect noqa:ARG002 on run(params): params IS used via
-  > operations_for(params) in _resolve_specs.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **maintenance:** iter-1 SOLID — OperationDeclaration protocol (LSP/OCP)<br>
-  > Introduce OperationDeclaration as a @runtime_checkable Protocol that both
-  > MaintainTable and MaintainSchema implement, replacing the isinstance branch
-  > in _resolve_specs with a uniform op.resolve(config) call.
-  > _protocol.py: add OperationDeclaration Protocol
-  > _builder.py: add resolve() to MaintainTable (returns [_to_spec()]) and
-  > MaintainSchema (delegates to _expand)
-  > _step.py: operations ClassVar typed as list[OperationDeclaration];
-  > __init_subclass__ validates via isinstance(op, OperationDeclaration)
-  > _runner.py: _resolve_specs simplified to a single list comprehension
-  > __init__.py: export OperationDeclaration
-  > Fixes LSP (MaintainTable/MaintainSchema now substitutable) and OCP partial
-  > (new operation declaration types no longer require touching the runner).
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **maintenance:** iter-2 SOLID — centralize duplication + reduce CC (SRP/DRY)<br>
-  > _builder.py: extract _expand_for_schemas() pure function — single source of
-  > truth for schema-prefix filtering. MaintainSchema._expand delegates to it.
-  > _runner.py: run_from_config calls _expand_for_schemas instead of reimplementing
-  > the loop; extract _resolve_location(table_ref) -> TableLocation | None to
-  > encapsulate the try/except + missing_table_policy logic. _run_one CC drops
-  > from 7 to 3 (early-return on None + single try block).
-  > backends/_delta_rs.py: extract _open(uri, location) to consolidate DeltaTable
-  > + WriterProperties + CommitProperties construction repeated in all three methods.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **maintenance:** iter-3 SOLID — polymorphic OpSpec dispatch (OCP)<br>
-  > Eliminate the if-chain in _run_one and make the module extensible to new
-  > operation types (RESTORE, CHECKPOINT…) without touching runner or builders.
-  > Design:
-  > _ops.py: VacuumSpec/CompactSpec/ZOrderSpec each gain a ClassVar name and
-  > execute(maintainer, uri, location) method — self-dispatching Strategy pattern.
-  > MaintenanceSpec replaces three nullable fields with ops: tuple[OpSpec, ...].
-  > _protocol.py: add OpSpec @runtime_checkable Protocol (name + execute).
-  > TableMaintenanceResult switches to op_results: dict[str, result] with
-  > @property vacuum/compact/z_order accessors for backwards compatibility.
-  > _builder.py: internal _ops list replaces three separate fields; _to_spec /
-  > _expand build the tuple. _expand_for_schemas takes an ops tuple instead of
-  > three params — one call site, one contract.
-  > _runner.py: _run_one becomes a single loop:
-  > for op in spec.ops: op_results[op.name] = op.execute(maintainer, uri, loc)
-  > Zero if-chain; adding a new OpSpec type requires zero changes here.
-  > To add a new operation (e.g. RESTORE):
-  > 1. Create RestoreSpec(name="restore") with execute() → RestoreResult
-  > 2. Add .restore() fluent method to MaintainTable / MaintainSchema
-  > 3. Done — runner, protocol, and tests need no changes.
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-- **maintenance:** post-iter cleanup — OpSpec ClassVar + deduplicate validation<br>
-  > _ops.py: MaintenanceSpec.ops typed as tuple[OpSpec, ...] for consistency
-  > with the runner loop; import OpSpec under TYPE_CHECKING.
-  > _protocol.py: OpSpec.name declared as ClassVar[str] so mypy correctly
-  > validates VacuumSpec/CompactSpec/ZOrderSpec as structural subtypes.
-  > _builder.py: extract _assert_no_conflicting_ops() pure function — single
-  > source of truth for the compact⊕z_order invariant, called from _to_spec
-  > and _expand (was duplicated in 3 places).
-  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
-
-
-
-
-
-