loom-kernel 0.10.3__tar.gz → 0.10.4__tar.gz

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/CHANGELOG.md

@@ -1,3 +1,40 @@
+# 🚀 Release 0.10.4 ([#52](https://github.com/the-reacher-data/loom-py/pull/52)) ([`3bf88be`](https://github.com/the-reacher-data/loom-py/commit/3bf88bee9210893dbf9522c1cf5c0424c069e43e))
+
+
+## ✨ Features
+### storage
+- **storage:** add MissingTablePolicy.ERROR to block auto-creation<br>
+  > Tables must be pre-created via the catalog process. With schema_mode,
+  > steps using SchemaMode.OVERWRITE could silently create tables bypassing
+  > catalog governance. ERROR unconditionally blocks creation regardless of
+  > the step's schema_mode.
+  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+
+
+
+## 🐛 Fixes
+### runner
+- **runner:** cleanup RUN-scope checkpoints in ETLRunner.run() finally block<br>
+  > CheckpointScope.RUN promises cleanup 'in the finally block of every
+  > pipeline run' but ETLRunner.run() never called cleanup_run(). RUN-scope
+  > checkpoint files accumulated in S3 indefinitely after every run.
+  > Wrap flush_runner in its own try/finally so cleanup always executes
+  > even if the observability flush raises. Cleanup errors are caught and
+  > logged as warnings to avoid masking the original pipeline exception.
+  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+  > --------
+  > Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
+
+
+
+
+
+
+
+
+
+
+
 # 🚀 Release 0.10.3 ([#48](https://github.com/the-reacher-data/loom-py/pull/48)) ([`15c54d0`](https://github.com/the-reacher-data/loom-py/commit/15c54d09d74297bac63508a11b23ec5048b8128a))
 
 

loom_kernel-0.10.4/CHANGELOG_RELEASE.md

@@ -0,0 +1,35 @@
+# 🚀 Release 0.10.4 ([#52](https://github.com/the-reacher-data/loom-py/pull/52)) ([`3bf88be`](https://github.com/the-reacher-data/loom-py/commit/3bf88bee9210893dbf9522c1cf5c0424c069e43e))
+
+
+## ✨ Features
+### storage
+- **storage:** add MissingTablePolicy.ERROR to block auto-creation<br>
+  > Tables must be pre-created via the catalog process. With schema_mode,
+  > steps using SchemaMode.OVERWRITE could silently create tables bypassing
+  > catalog governance. ERROR unconditionally blocks creation regardless of
+  > the step's schema_mode.
+  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+
+
+
+## 🐛 Fixes
+### runner
+- **runner:** cleanup RUN-scope checkpoints in ETLRunner.run() finally block<br>
+  > CheckpointScope.RUN promises cleanup 'in the finally block of every
+  > pipeline run' but ETLRunner.run() never called cleanup_run(). RUN-scope
+  > checkpoint files accumulated in S3 indefinitely after every run.
+  > Wrap flush_runner in its own try/finally so cleanup always executes
+  > even if the observability flush raises. Cleanup errors are caught and
+  > logged as warnings to avoid masking the original pipeline exception.
+  > Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+  > --------
+  > Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
+
+
+
+
+
+
+
+
+

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loom-kernel
-Version: 0.10.3
+Version: 0.10.4
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Typing :: Typed

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "loom-kernel"
-version = "0.10.3"
+version = "0.10.4"
 description = "Loom Python project"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -268,7 +268,7 @@ markers = [
 
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.10.3"
+version = "0.10.4"
 tag_format = "v$version"
 version_files = [
     "pyproject.toml:project.version",

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/src/loom/etl/backends/_write_policy.py

@@ -62,6 +62,8 @@ def _can_create_missing_table(
     missing_table_policy: MissingTablePolicy,
 ) -> bool:
     """Return ``True`` when table creation is allowed for missing destination."""
+    if missing_table_policy is MissingTablePolicy.ERROR:
+        return False
     return missing_table_policy is MissingTablePolicy.CREATE or schema_mode is SchemaMode.OVERWRITE
 
 

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/src/loom/etl/runner/core.py

@@ -199,7 +199,16 @@ class ETLRunner:
         try:
             self._executor.run_pipeline(plan, params, ctx)
         finally:
-            flush_runner(self)
+            try:
+                flush_runner(self)
+            finally:
+                if self._checkpoint_store is not None:
+                    try:
+                        self._checkpoint_store.cleanup_run(ctx.run_id)
+                    except Exception:
+                        _log.warning(
+                            "checkpoint cleanup failed run_id=%s", ctx.run_id, exc_info=True
+                        )
 
     def flush(self) -> None:
         """Flush buffered ETL observability sinks after a run."""

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/src/loom/etl/storage/_config.py

@@ -34,6 +34,7 @@ class MissingTablePolicy(StrEnum):
 
     SCHEMA_MODE = "schema_mode"
     CREATE = "create"
+    ERROR = "error"
 
 
 class CatalogConnection(LoomFrozenStruct, frozen=True):

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/tests/unit/etl/backends/test_polars/test_backend.py

@@ -242,6 +242,15 @@ def test_writer_append_first_write_requires_create_policy(tmp_path: Path) -> Non
         writer.append(frame, TableRef("staging.append_first"), None)
 
 
+def test_writer_error_policy_blocks_creation_even_with_overwrite_schema(tmp_path: Path) -> None:
+    writer = PolarsTargetWriter(tmp_path, missing_table_policy=MissingTablePolicy.ERROR)
+    frame = pl.DataFrame({"id": [1], "v": [10.0]}).lazy()
+    spec = ReplaceSpec(table_ref=TableRef("staging.must_exist"), schema_mode=SchemaMode.OVERWRITE)
+
+    with pytest.raises(SchemaNotFoundError):
+        writer.write(frame, spec, None)
+
+
 def test_writer_append_creates_table_on_first_write(tmp_path: Path) -> None:
     writer = PolarsTargetWriter(tmp_path, missing_table_policy=MissingTablePolicy.CREATE)
     frame = pl.DataFrame({"id": [1], "v": [10.0]}).lazy()

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/tests/unit/etl/backends/test_write_policy_historify.py

@@ -233,6 +233,15 @@ class TestHistorifyMissingTablePolicy:
         with pytest.raises(SchemaNotFoundError):
             writer.write([1], spec, None)
 
+    def test_missing_table_with_error_policy_raises_regardless_of_schema_mode(self) -> None:
+        for schema_mode in (SchemaMode.OVERWRITE, SchemaMode.STRICT):
+            writer = _StubWritePolicy(
+                schema_exists=False,
+                missing_table_policy=MissingTablePolicy.ERROR,
+            )
+            with pytest.raises(SchemaNotFoundError):
+                writer.write([1], _spec(schema_mode=schema_mode), None)
+
     def test_existing_table_always_delegates_to_historify(self) -> None:
         writer = _StubWritePolicy(schema_exists=True)
         writer.write([1], _spec(), None)

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/tests/unit/etl/test_runner.py

@@ -470,3 +470,90 @@ class TestRunnerCleaner:
         assert runner._checkpoint_store is not None
         runner._checkpoint_store.cleanup_run("run-xyz")
         assert any("run-xyz" in path for path in deleted)
+
+    def test_run_cleans_up_run_scope_checkpoints_on_success(self, tmp_path: Path) -> None:
+        import polars as pl
+        from deltalake import write_deltalake
+
+        from loom.etl.storage._config import MissingTablePolicy
+
+        deleted: list[str] = []
+
+        class SpyCleaner:
+            def delete_tree(self, path: str) -> None:
+                deleted.append(path)
+
+        src = tmp_path / "raw" / "items"
+        src.mkdir(parents=True)
+        write_deltalake(str(src), pl.DataFrame({"id": [1, 2]}), mode="overwrite")
+
+        config = StorageConfig(
+            defaults=StorageDefaults(table_path=TablePathConfig(uri=str(tmp_path))),
+            missing_table_policy=MissingTablePolicy.CREATE,
+            temp=TempConfig(root="s3://fake-bucket/checkpoints"),
+        )
+        runner = ETLRunner.from_config(config, cleaner=SpyCleaner())
+
+        class _P(ETLParams):
+            pass
+
+        class _Step(ETLStep[_P]):
+            items = FromTable("raw.items")
+            target = IntoTable("staging.out").replace()
+
+            def execute(self, params: _P, *, items: Any) -> Any:  # type: ignore[override]
+                return items
+
+        class _Proc(ETLProcess[_P]):
+            steps = [_Step]
+
+        class _Pipeline(ETLPipeline[_P]):
+            processes = [_Proc]
+
+        runner.run(_Pipeline, _P(), run_id="test-run-abc")
+
+        assert any("test-run-abc" in path for path in deleted)
+
+    def test_run_cleans_up_run_scope_checkpoints_on_failure(self, tmp_path: Path) -> None:
+        import polars as pl
+        from deltalake import write_deltalake
+
+        from loom.etl.storage._config import MissingTablePolicy
+
+        deleted: list[str] = []
+
+        class SpyCleaner:
+            def delete_tree(self, path: str) -> None:
+                deleted.append(path)
+
+        src = tmp_path / "raw" / "items"
+        src.mkdir(parents=True)
+        write_deltalake(str(src), pl.DataFrame({"id": [1]}), mode="overwrite")
+
+        config = StorageConfig(
+            defaults=StorageDefaults(table_path=TablePathConfig(uri=str(tmp_path))),
+            missing_table_policy=MissingTablePolicy.CREATE,
+            temp=TempConfig(root="s3://fake-bucket/checkpoints"),
+        )
+        runner = ETLRunner.from_config(config, cleaner=SpyCleaner())
+
+        class _P(ETLParams):
+            pass
+
+        class _FailStep(ETLStep[_P]):
+            items = FromTable("raw.items")
+            target = IntoTable("staging.fail").replace()
+
+            def execute(self, params: _P, *, items: Any) -> Any:  # type: ignore[override]
+                raise RuntimeError("intentional failure")
+
+        class _Proc(ETLProcess[_P]):
+            steps = [_FailStep]
+
+        class _Pipeline(ETLPipeline[_P]):
+            processes = [_Proc]
+
+        with pytest.raises(RuntimeError, match="intentional failure"):
+            runner.run(_Pipeline, _P(), run_id="fail-run-xyz")
+
+        assert any("fail-run-xyz" in path for path in deleted)

{loom_kernel-0.10.3 → loom_kernel-0.10.4}/uv.lock

@@ -1583,7 +1583,7 @@ wheels = [
 
 [[package]]
 name = "loom-kernel"
-version = "0.10.3"
+version = "0.10.4"
 source = { editable = "." }
 dependencies = [
     { name = "aiocache" },

loom_kernel-0.10.3/CHANGELOG_RELEASE.md

@@ -1,50 +0,0 @@
-# 🚀 Release 0.10.3 ([#48](https://github.com/the-reacher-data/loom-py/pull/48)) ([`15c54d0`](https://github.com/the-reacher-data/loom-py/commit/15c54d09d74297bac63508a11b23ec5048b8128a))
-
-
-## ✨ Features
-### polars
-- **polars:** streaming Arrow writes for replace_partitions<br>
-  > Adds a target-level streaming flag on AppendSpec/ReplaceSpec/
-  > ReplacePartitionsSpec/ReplaceWhereSpec (UpsertSpec excluded — MERGE has
-  > no streaming source path in delta-rs). The Polars backend honours the
-  > flag today for replace_partitions against existing tables: the
-  > LazyFrame is sunk to a lz4-compressed IPC spool and re-opened as a
-  > pyarrow.RecordBatchReader passed straight to write_deltalake, bounding
-  > peak RAM to roughly one batch + delta-rs internals.
-  > Partition predicate is computed via a projection-pushdown scan over the
-  > same spool. Schema alignment and the Null-dtype guard run lazily on the
-  > LazyFrame before the sink. Other modes accept the flag for forward
-  > compatibility but still materialise the frame.
-  > Spool directory is configurable via LOOM_SPOOL_DIR so container
-  > workloads with a tmpfs /tmp (Fargate) can spill to a real disk.
-  > Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
-
-
-
-## 🐛 Fixes
-### clickhouse
-- **clickhouse:** compress streaming IPC spool with lz4<br>
-  > The read_streaming path spooled Arrow IPC files without compression,
-  > producing ~10-15 GB temporary files for large ClickHouse result sets
-  > (e.g. 19M-row CDC tables). Combined with a second lz4-compressed spool
-  > from the downstream streaming Delta write, total ephemeral disk usage
-  > exceeded 20 GB on Fargate tasks.
-  > Adding lz4 compression to the CH spool reduces it ~3x (~4-5 GB),
-  > bringing the combined footprint to ~8-10 GB — within the Fargate
-  > default ephemeral limit without requiring task definition changes.
-  > Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
-
-
-
-
-
-
-
-
-
-## 🔖 Other
-- chore(version) Update version<br>
-  > --------
-  > Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
-
-