loom-kernel 0.3.0__tar.gz → 0.4.0__tar.gz

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/.github/workflows/release.yml

@@ -99,23 +99,28 @@ jobs:
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
 
           BRANCH="docs/release-v${VERSION}"
-          git checkout -b "${BRANCH}"
+          git checkout -B "${BRANCH}"
 
           if [ -f CHANGELOG_RELEASE.md ]; then
-            cat CHANGELOG_RELEASE.md CHANGELOG.md > CHANGELOG_MERGED.md
-            mv CHANGELOG_MERGED.md CHANGELOG.md
+            HEADER=$(head -1 CHANGELOG_RELEASE.md)
+            if ! grep -qF "${HEADER}" CHANGELOG.md; then
+              cat CHANGELOG_RELEASE.md CHANGELOG.md > CHANGELOG_MERGED.md
+              mv CHANGELOG_MERGED.md CHANGELOG.md
+            fi
           fi
 
           git add pyproject.toml CHANGELOG.md
           git diff --staged --quiet && exit 0
           git commit -m "chore(release): bump version to ${VERSION}"
-          git push origin "${BRANCH}"
-
-          gh pr create \
-            --title "chore(release): bump version to ${VERSION}" \
-            --body "Automated version bump and changelog update after release v${VERSION}." \
-            --base master \
-            --head "${BRANCH}"
+          git push origin "${BRANCH}" --force
+
+          if ! gh pr list --head "${BRANCH}" --state open --json number --jq '.[0].number' | grep -qE '^[0-9]+$'; then
+            gh pr create \
+              --title "chore(release): bump version to ${VERSION}" \
+              --body "Automated version bump and changelog update after release v${VERSION}." \
+              --base master \
+              --head "${BRANCH}"
+          fi
 
           gh pr merge "${BRANCH}" --squash --auto --delete-branch
 

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/.gitignore

@@ -19,6 +19,7 @@ htmlcov/
 docs/__pycache__/
 .claude/
 .claude.md
+CLAUDE.md
 AGENTS.md
 
 # Internal discussion docs (not part of distributable source)

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/.readthedocs.yaml

@@ -13,3 +13,9 @@ python:
     - requirements: docs/requirements.txt
     - method: pip
       path: .
+      extra_requirements:
+        - rest
+        - sqlalchemy
+        - cache
+        - config
+        - prometheus

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/CHANGELOG.md

@@ -1,61 +1,18 @@
-# 🚀 Release 0.3.0 ([#14](https://github.com/the-reacher-data/loom-py/pull/14)) ([`ef414c5`](https://github.com/the-reacher-data/loom-py/commit/ef414c5bfd303296af450840318dfbe9d301e5d1))
+# 🚀 Release 0.4.0 ([#18](https://github.com/the-reacher-data/loom-py/pull/18)) ([`09e5aa3`](https://github.com/the-reacher-data/loom-py/commit/09e5aa340f88e0488daca2bf10320aad7aebbe1d))
 
 
 ## ✨ Features
-### config
-- **config:** add cloud URI support and pluggable resolver extension point<br>
-  > Add fsspec as a hard dependency of loom[config]
-  > load_config() now accepts s3://, gs://, abfss://, r2:// URIs via fsspec
-  > Add ConfigResolver protocol for pluggable ${prefix:key} resolution at
-  > parse time (enables SSM, Key Vault, etc. without baking secrets into images)
-  > Resolver registration is idempotent; resolvers are evaluated at job startup
-  > so secret rotation takes effect on the next run
-  > Migrate loom.etl.runner.config_loader to use core load_config, removing
-  > the parallel OmegaConf implementation
-  > ETL _load_yaml inherits cloud URI and resolver support transparently
-
-
 ### etl
-- **etl:** add FileLocator with explicit alias API for file routes<br>
-  > Introduces `FileLocator` protocol and `MappingFileLocator` so that
-  > `FromFile.alias("name")` / `IntoFile.alias("name")` specs resolve at
-  > runtime through `storage.files` config rather than hard-coded URIs.
-  > `FileLocation` / `FileLocator` / `MappingFileLocator` in `storage/_file_locator.py`
-  > `StorageConfig.to_file_locator()` returns `MappingFileLocator | None`
-  > (None when `files` is empty — no conditional needed at call sites)
-  > `FromFile.alias()` / `IntoFile.alias()` classmethods set `is_alias=True`
-  > on the emitted spec
-  > `is_alias: bool` added to `FileSourceSpec` and `FileSpec`
-  > Polars and Spark backends resolve aliases via injected `file_locator`
-  > Both providers wired: `file_locator = config.to_file_locator()`
-  > Full test coverage across io, storage, and backend layers
+- **etl:** add IntoHistory builder and SCD Type 2 domain contracts
 
 
 
-## 🐛 Fixes
-### observability
-- **observability:** honor missing table policy for record store writers
 
 
-## 📖 Documentation
-### etl
-- **etl:** keep only user guide and drop refactor docs
-- **etl:** expand ETL documentation and update directory table<br>
-  > Add dummy-loom-etl companion repo link in README and etl guide
-  > Expand README subpaths table with loom.etl and loom.core.config entries
-  > Add FileLocator/alias API, cloud config URI, and ConfigResolver sections to etl guide
-  > Add loom.etl.backends (polars + spark) to etl.rst API reference
-  > Add loom.core.config to core.rst API reference
 
 
 
 
-## ♻️ Refactor
-### stepsql
-- **stepsql:** delegate SQL execution to backend readers
-
-
-
 
 
 # 🚀 Release 0.3.0 ([#14](https://github.com/the-reacher-data/loom-py/pull/14)) ([`ef414c5`](https://github.com/the-reacher-data/loom-py/commit/ef414c5bfd303296af450840318dfbe9d301e5d1))
@@ -118,8 +75,6 @@
 
 
 
-
-
 # 🚀 Release 0.2.1 ([#12](https://github.com/the-reacher-data/loom-py/pull/12)) ([`87f7d1f`](https://github.com/the-reacher-data/loom-py/commit/87f7d1f1eb1ccde71d0aca1c5584b83317e30707))
 
 ## ✨ Features

loom_kernel-0.4.0/CHANGELOG_RELEASE.md

@@ -0,0 +1,15 @@
+# 🚀 Release 0.4.0 ([#18](https://github.com/the-reacher-data/loom-py/pull/18)) ([`09e5aa3`](https://github.com/the-reacher-data/loom-py/commit/09e5aa340f88e0488daca2bf10320aad7aebbe1d))
+
+
+## ✨ Features
+### etl
+- **etl:** add IntoHistory builder and SCD Type 2 domain contracts
+
+
+
+
+
+
+
+
+

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loom-kernel
-Version: 0.3.0
+Version: 0.4.0
 Summary: Loom Python project
 License-File: LICENSE
 Requires-Python: >=3.11
@@ -78,7 +78,8 @@ Framework-agnostic Python toolkit to build backend applications with:
 - repositories decoupled from infrastructure
 - REST/FastAPI adapters with OpenAPI generation
 - background jobs and Celery workers, first-class
-- testing utilities for business workflows
+- **declarative ETL** — compile-time-validated pipelines for Polars and Spark
+- testing utilities for business workflows and ETL steps
 
 ## Purpose
 
@@ -452,6 +453,66 @@ For deeper references, review the integration examples under
 For a runnable full-stack sample with all patterns combined, check the companion repository
 [`dummy-loom`](https://github.com/the-reacher-data/dummy-loom).
 
+## ETL quick start
+
+Install a backend:
+
+```bash
+pip install "loom-kernel[etl-polars]"
+# or
+pip install "loom-kernel[etl-spark]"
+```
+
+Declare a pipeline — sources, targets, and transformation logic are explicit and compile-time validated:
+
+```python
+from datetime import date
+import polars as pl
+from loom.etl import ETLParams, ETLStep, ETLProcess, ETLPipeline, ETLRunner, FromTable, IntoTable
+
+class DailyParams(ETLParams):
+    run_date: date
+
+class CleanOrders(ETLStep[DailyParams]):
+    orders = FromTable("raw.orders").columns("id", "amount", "run_date")
+    target = IntoTable("staging.orders").replace()
+
+    def execute(self, params: DailyParams, *, orders: pl.LazyFrame) -> pl.LazyFrame:
+        return orders.filter(pl.col("amount") > 0)
+
+class DailyProcess(ETLProcess[DailyParams]):
+    steps = [CleanOrders]
+
+class DailyPipeline(ETLPipeline[DailyParams]):
+    processes = [DailyProcess]
+
+runner = ETLRunner.from_yaml("config/etl.yaml")
+runner.run(DailyPipeline, DailyParams(run_date=date.today()))
+```
+
+Write modes — `replace`, `append`, `replace_partition`, `replace_partitions`, `replace_where`, `upsert` — are declared on the target, validated at compile time. Partition predicates use the `params` proxy so no values are hard-coded:
+
+```python
+from loom.etl import params
+
+target = IntoTable("staging.orders").replace_partition(
+    year=params.run_date.year,
+    month=params.run_date.month,
+)
+```
+
+File aliases decouple paths from pipelines — declare once in YAML, reference by name:
+
+```python
+events = FromFile.alias("events_raw", format=Format.CSV)
+target = IntoFile.alias("exports_daily", format=Format.PARQUET)
+```
+
+For a full Polars + Spark + Delta Lake example see [`dummy-loom-etl`](https://github.com/the-reacher-data/dummy-loom-etl).
+For the complete API reference and write-mode guide see the [ETL docs](https://loom-py.readthedocs.io/en/latest/guides/etl.html).
+
+---
+
 ## Performance
 
 `loom-kernel` adds zero measurable overhead at the concurrency levels typical of

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/README.md

@@ -20,7 +20,8 @@ Framework-agnostic Python toolkit to build backend applications with:
 - repositories decoupled from infrastructure
 - REST/FastAPI adapters with OpenAPI generation
 - background jobs and Celery workers, first-class
-- testing utilities for business workflows
+- **declarative ETL** — compile-time-validated pipelines for Polars and Spark
+- testing utilities for business workflows and ETL steps
 
 ## Purpose
 
@@ -394,6 +395,66 @@ For deeper references, review the integration examples under
 For a runnable full-stack sample with all patterns combined, check the companion repository
 [`dummy-loom`](https://github.com/the-reacher-data/dummy-loom).
 
+## ETL quick start
+
+Install a backend:
+
+```bash
+pip install "loom-kernel[etl-polars]"
+# or
+pip install "loom-kernel[etl-spark]"
+```
+
+Declare a pipeline — sources, targets, and transformation logic are explicit and compile-time validated:
+
+```python
+from datetime import date
+import polars as pl
+from loom.etl import ETLParams, ETLStep, ETLProcess, ETLPipeline, ETLRunner, FromTable, IntoTable
+
+class DailyParams(ETLParams):
+    run_date: date
+
+class CleanOrders(ETLStep[DailyParams]):
+    orders = FromTable("raw.orders").columns("id", "amount", "run_date")
+    target = IntoTable("staging.orders").replace()
+
+    def execute(self, params: DailyParams, *, orders: pl.LazyFrame) -> pl.LazyFrame:
+        return orders.filter(pl.col("amount") > 0)
+
+class DailyProcess(ETLProcess[DailyParams]):
+    steps = [CleanOrders]
+
+class DailyPipeline(ETLPipeline[DailyParams]):
+    processes = [DailyProcess]
+
+runner = ETLRunner.from_yaml("config/etl.yaml")
+runner.run(DailyPipeline, DailyParams(run_date=date.today()))
+```
+
+Write modes — `replace`, `append`, `replace_partition`, `replace_partitions`, `replace_where`, `upsert` — are declared on the target, validated at compile time. Partition predicates use the `params` proxy so no values are hard-coded:
+
+```python
+from loom.etl import params
+
+target = IntoTable("staging.orders").replace_partition(
+    year=params.run_date.year,
+    month=params.run_date.month,
+)
+```
+
+File aliases decouple paths from pipelines — declare once in YAML, reference by name:
+
+```python
+events = FromFile.alias("events_raw", format=Format.CSV)
+target = IntoFile.alias("exports_daily", format=Format.PARQUET)
+```
+
+For a full Polars + Spark + Delta Lake example see [`dummy-loom-etl`](https://github.com/the-reacher-data/dummy-loom-etl).
+For the complete API reference and write-mode guide see the [ETL docs](https://loom-py.readthedocs.io/en/latest/guides/etl.html).
+
+---
+
 ## Performance
 
 `loom-kernel` adds zero measurable overhead at the concurrency levels typical of

{loom_kernel-0.3.0 → loom_kernel-0.4.0}/docs/conf.py

@@ -29,7 +29,7 @@ extensions = [
 ]
 
 templates_path = ["_templates"]
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "guides", "architecture", "examples-repo"]
 
 html_theme = "furo"
 html_static_path = ["_static"]

loom_kernel-0.4.0/docs/etl/examples.md

@@ -0,0 +1,29 @@
+# ETL Examples
+
+The companion repository [`dummy-loom-etl`](https://github.com/the-reacher-data/dummy-loom-etl)
+contains runnable end-to-end examples of `loom.etl` with both Polars and Spark
+backends.
+
+## What it covers
+
+- Polars and Spark pipeline declarations
+- Delta Lake read/write modes (`replace`, `upsert`, `replace_partitions`)
+- Cloud storage configuration (S3, GCS, Azure) via `fsspec`
+- File aliases with `FromFile.alias` / `IntoFile.alias`
+- Observability and run sinks
+- Integration tests with `PolarsStepRunner` and `SparkTestSession`
+
+## Quick start
+
+```bash
+git clone https://github.com/the-reacher-data/dummy-loom-etl
+cd dummy-loom-etl
+make test      # run Polars and Spark test suites
+make run       # run the sample pipeline locally
+```
+
+## Next steps
+
+- [ETL pipelines guide](pipelines.md)
+- [ETL testing guide](testing.md)
+- [API reference](../reference/api/etl)

loom_kernel-0.4.0/docs/etl/testing.md

@@ -0,0 +1,65 @@
+# ETL Testing
+
+`loom.etl.testing` provides in-memory harnesses for testing `ETLStep`
+subclasses without spinning up a real catalog or writing to Delta Lake.
+
+## Test a single step with Polars
+
+`PolarsStepRunner` seeds source tables as plain Python tuples, runs one step,
+and returns a `StepResult` for assertions.
+
+```python
+from loom.etl.testing import PolarsStepRunner
+
+def test_clean_orders():
+    runner = PolarsStepRunner()
+    runner.seed("raw.orders", [(1, 10.0), (2, -5.0)], ["id", "amount"])
+    result = runner.run(CleanOrdersStep, NoParams())
+
+    result.assert_count(1)
+    result.assert_schema({"id": "Int64", "amount": "Float64"})
+```
+
+`StepResult` methods: `assert_schema`, `assert_count`, `assert_not_empty`,
+`show`, `to_polars`.
+
+## Reusable seed datasets with `ETLScenario`
+
+`ETLScenario` stores seed data as plain tuples so the same dataset can be used
+with any backend runner.
+
+```python
+from loom.etl.testing import ETLScenario
+
+ORDERS = (
+    ETLScenario()
+    .with_table("raw.orders", [(1, 10.0), (2, 20.0)], ["id", "amount"])
+)
+
+def test_double_amount(loom_polars_runner):
+    ORDERS.apply(loom_polars_runner)
+    result = loom_polars_runner.run(DoubleAmountStep, NoParams())
+    result.assert_count(2)
+```
+
+Pytest fixtures auto-registered: `loom_polars_runner`, `loom_spark_runner`.
+
+## Spark integration tests
+
+Use `SparkTestSession` for a real PySpark + Delta session:
+
+```python
+from loom.etl.testing.spark import SparkTestSession
+
+with SparkTestSession.start() as spark:
+    runner = SparkStepRunner(spark)
+    runner.seed("raw.orders", [(1, 10.0)], ["id", "amount"])
+    result = runner.run(CleanOrdersStep, NoParams())
+    result.assert_count(1)
+```
+
+## Compile-time stubs
+
+- `StubCatalog` — in-memory catalog.
+- `StubSourceReader` / `StubTargetWriter` — in-memory I/O.
+- `StubRunObserver` — records pipeline lifecycle events.

loom_kernel-0.4.0/docs/getting-started/etl.md

@@ -0,0 +1,136 @@
+# ETL Quickstart
+
+`loom.etl` is a declarative ETL subsystem with compile-time validation,
+backend-agnostic declarations, and a single runtime entrypoint (`ETLRunner`).
+
+## Install
+
+Choose one backend:
+
+```bash
+pip install "loom-kernel[etl-polars]"
+# or
+pip install "loom-kernel[etl-spark]"
+```
+
+## Minimal pipeline
+
+```python
+from datetime import date
+
+import polars as pl
+from loom.etl import (
+    ETLParams,
+    ETLStep,
+    ETLProcess,
+    ETLPipeline,
+    ETLRunner,
+    FromTable,
+    IntoTable,
+)
+
+
+class DailyParams(ETLParams):
+    run_date: date
+
+
+class CleanOrders(ETLStep[DailyParams]):
+    orders = FromTable("raw.orders").columns("id", "amount", "run_date")
+    target = IntoTable("staging.orders").replace()
+
+    def execute(self, params: DailyParams, *, orders: pl.LazyFrame) -> pl.LazyFrame:
+        return orders.filter(pl.col("amount") > 0)
+
+
+class DailyProcess(ETLProcess[DailyParams]):
+    steps = [CleanOrders]
+
+
+class DailyPipeline(ETLPipeline[DailyParams]):
+    processes = [DailyProcess]
+
+
+runner = ETLRunner.from_dict(
+    storage={
+        "engine": "polars",
+        "defaults": {"table_path": {"uri": "/var/lib/loom/lake"}},
+    }
+)
+runner.run(DailyPipeline, DailyParams(run_date=date(2026, 3, 30)))
+```
+
+## Basic write modes
+
+Every `IntoTable` target declares exactly one write mode by chaining a method.
+
+| Mode | What it does | Example |
+|------|--------------|---------|
+| `append` | Add rows to the table | `IntoTable("staging.orders").append()` |
+| `replace` | Full overwrite | `IntoTable("staging.orders").replace()` |
+| `replace_partitions` | Overwrite only partitions present in the batch | `IntoTable("staging.orders").replace_partitions("year", "month")` |
+| `replace_partition` | Overwrite a single known partition | `IntoTable("staging.orders").replace_partition(year=params.run_date.year)` |
+| `replace_where` | Overwrite rows matching a predicate | `IntoTable("staging.orders").replace_where(col("date") == params.run_date)` |
+| `upsert` | Merge on key columns | `IntoTable("staging.orders").upsert(keys=("order_id",))` |
+
+See the [ETL pipelines guide](../etl/pipelines.md) for the full write-mode reference.
+
+## YAML config
+
+```yaml
+storage:
+  engine: polars
+  defaults:
+    table_path:
+      uri: s3://my-lake
+      storage_options:
+        AWS_REGION: ${oc.env:AWS_REGION}
+        AWS_ACCESS_KEY_ID: ${oc.env:AWS_ACCESS_KEY_ID}
+        AWS_SECRET_ACCESS_KEY: ${oc.env:AWS_SECRET_ACCESS_KEY}
+
+  tmp_root: /var/lib/loom/lake/_tmp
+
+observability:
+  log: true
+  slow_step_threshold_ms: 30000
+  run_sink:
+    root: /var/lib/loom/lake/_runs
+```
+
+```python
+from loom.etl import ETLRunner
+
+runner = ETLRunner.from_yaml("config/etl.yaml")
+```
+
+## File aliases
+
+Hard-coding file paths couples logic to infrastructure. Use aliases instead:
+
+```yaml
+storage:
+  engine: polars
+  files:
+    - name: events_raw
+      path:
+        uri: s3://raw-bucket/events/
+        storage_options:
+          AWS_REGION: eu-west-1
+```
+
+```python
+from loom.etl import ETLStep, FromFile, IntoFile, Format
+
+class LoadEvents(ETLStep[DailyParams]):
+    events = FromFile.alias("events_raw", format=Format.CSV)
+    target = IntoFile.alias("exports_daily", format=Format.PARQUET)
+
+    def execute(self, params: DailyParams, *, events: pl.LazyFrame) -> pl.LazyFrame:
+        return events
+```
+
+## Next steps
+
+- [ETL pipelines guide](../etl/pipelines.md) — full write modes, Spark runtime, cloud config, and pluggable resolvers
+- [ETL testing guide](../etl/testing.md) — in-memory runners, scenarios, and stubs
+- [ETL examples](../etl/examples.md) — companion repository with runnable Polars and Spark pipelines
+- [API reference](../reference/api/etl)