medallantic 0.27.0__tar.gz

medallantic-0.27.0/.gitignore

@@ -0,0 +1,35 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+.eggs/
+dist/
+build/
+site/
+.venv/
+venv/
+.env
+
+# Tools
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+# macOS
+.DS_Store
+**/.DS_Store
+._*
+**/._*
+
+examples/_file_storage_out/
+examples/_generated_*.py
+tmp_cli_contracts/
+tmp_/
+
+# Local durable workspace (reports, artifacts, schema history)
+.etlantic/
+
+# Local authoring artifacts (not part of the repo)
+pipeline.definition.json

medallantic-0.27.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: medallantic
+Version: 0.27.0
+Summary: Engine-agnostic medallion pipelines built on ETLantic.
+Project-URL: Homepage, https://github.com/eddiethedean/etlantic
+Project-URL: Documentation, https://github.com/eddiethedean/etlantic/tree/main/docs
+Project-URL: Repository, https://github.com/eddiethedean/etlantic
+Project-URL: Issues, https://github.com/eddiethedean/etlantic/issues
+Project-URL: Changelog, https://github.com/eddiethedean/etlantic/blob/main/CHANGELOG.md
+Author-email: Odo Matthews <odosmatthews@gmail.com>
+License-Expression: MIT
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: etlantic<0.28,>=0.27.0
+Description-Content-Type: text/markdown
+
+# Medallantic
+
+Engine-agnostic medallion pipelines built on ETLantic.
+
+Medallantic owns bronze/silver/gold authoring and conventions. ETLantic owns
+the portable contracts, graph, validation, planning, execution lifecycle, and
+plugin coordination underneath it. **ETLantic core never gains medallion
+types.**
+
+The current release is the renamed SparkForge migration adapter. It maps
+SparkForge IR onto ordinary ETLantic `Extract` / `Step` / `Load`, `Profile`,
+`RunSelection` / `RunIntent`, and `PipelineRunReport` surfaces. A native
+engine-agnostic builder and live Spark/SQL parity are planned; see the
+[roadmap](ROADMAP.md).
+
+Documentation:
+
+- [Documentation index](docs/README.md)
+- [Getting started](docs/getting-started.md)
+- [SparkForge migration](docs/sparkforge-migration.md)
+- [Compatibility](docs/compatibility.md)
+- [Architecture](docs/architecture.md)
+
+## Install
+
+```bash
+pip install 'etlantic==0.27.0' 'medallantic==0.27.0'
+# or
+pip install 'etlantic[medallantic]'
+```
+
+The shipped adapter is **IR-only**: feed `SparkForgePipelineSpec` (JSON/YAML
+fixtures or hand-built dataclasses). There is **no** live
+`pipeline_builder` / SparkForge Python API bridge in this release.
+
+The adapter is registered explicitly by importing `medallantic` and
+calling its conversion helpers. The adapted result supplies an ordinary
+ETLantic pipeline and profile; select execution plugins such as
+`Profile.spark_engine="pyspark"` separately. Production profiles must
+allowlist every trusted execution plugin.
+
+## Quick start (IR → Pipeline)
+
+```python
+from medallantic import (
+    SparkForgePipelineSpec,
+    SparkForgeStepSpec,
+    StepKind,
+    LayerKind,
+    adapt_pipeline,
+    debug_request_from_sparkforge,
+    enrich_plan,
+)
+from etlantic.plan import plan_pipeline
+
+spec = SparkForgePipelineSpec(
+    name="ecommerce",
+    schema="demo",
+    steps=(
+        SparkForgeStepSpec(
+            name="orders",
+            kind=StepKind.BRONZE_RULES,
+            layer=LayerKind.BRONZE,
+            table_name="bronze_orders",
+        ),
+        SparkForgeStepSpec(
+            name="clean_orders",
+            kind=StepKind.SILVER_TRANSFORM,
+            layer=LayerKind.SILVER,
+            source="orders",
+            table_name="silver_orders",
+            write_mode="overwrite",
+        ),
+    ),
+)
+adapted = adapt_pipeline(spec)
+adapted.pipeline_cls.validate(profile=adapted.profile)
+plan = enrich_plan(
+    plan_pipeline(adapted.pipeline_cls, profile=adapted.profile),
+    adapted,
+)
+request = debug_request_from_sparkforge(mode="incremental", skip_writes=True)
+```
+
+## Delivery direction
+
+1. **Plan-only** — generate/inspect ETLantic plans from SparkForge IR
+   (`strict_delta=False` to warn instead of fail when Delta caps are unknown)
+2. **Dual reporting** — `adapt_run_result` → `PipelineRunReport`
+3. **ETLantic planning** — selections/intents via `debug_request_from_sparkforge`
+4. **Plugin execution** — `Profile.spark_engine="pyspark"` / SQL plugins
+5. **Native facade** — Medallantic owns medallion authoring and retires the
+   duplicated legacy engine-specific builders
+
+`transform_ref` / bronze `rules` emit `PMSF411` warnings: the adapter builds
+passthrough transforms for planning parity; it does not execute SparkForge
+callables. Write intents (including MERGE) are attached via `enrich_plan` for
+orchestration; the local runtime still gates materialization with
+`RunRequest.no_write`.
+
+See `docs/11_DEVELOPMENT/MIGRATION_0_9_TO_0_10.md`.
+
+## Boundary
+
+| Concern | Owner |
+|---|---|
+| bronze / silver / gold APIs | Medallantic |
+| portable graph, plan, reports | ETLantic |
+| legacy mapping + parity fixtures | Medallantic migration layer |
+| physical execution | ETLantic engine and storage plugins |

medallantic-0.27.0/README.md

@@ -0,0 +1,110 @@
+# Medallantic
+
+Engine-agnostic medallion pipelines built on ETLantic.
+
+Medallantic owns bronze/silver/gold authoring and conventions. ETLantic owns
+the portable contracts, graph, validation, planning, execution lifecycle, and
+plugin coordination underneath it. **ETLantic core never gains medallion
+types.**
+
+The current release is the renamed SparkForge migration adapter. It maps
+SparkForge IR onto ordinary ETLantic `Extract` / `Step` / `Load`, `Profile`,
+`RunSelection` / `RunIntent`, and `PipelineRunReport` surfaces. A native
+engine-agnostic builder and live Spark/SQL parity are planned; see the
+[roadmap](ROADMAP.md).
+
+Documentation:
+
+- [Documentation index](docs/README.md)
+- [Getting started](docs/getting-started.md)
+- [SparkForge migration](docs/sparkforge-migration.md)
+- [Compatibility](docs/compatibility.md)
+- [Architecture](docs/architecture.md)
+
+## Install
+
+```bash
+pip install 'etlantic==0.27.0' 'medallantic==0.27.0'
+# or
+pip install 'etlantic[medallantic]'
+```
+
+The shipped adapter is **IR-only**: feed `SparkForgePipelineSpec` (JSON/YAML
+fixtures or hand-built dataclasses). There is **no** live
+`pipeline_builder` / SparkForge Python API bridge in this release.
+
+The adapter is registered explicitly by importing `medallantic` and
+calling its conversion helpers. The adapted result supplies an ordinary
+ETLantic pipeline and profile; select execution plugins such as
+`Profile.spark_engine="pyspark"` separately. Production profiles must
+allowlist every trusted execution plugin.
+
+## Quick start (IR → Pipeline)
+
+```python
+from medallantic import (
+    SparkForgePipelineSpec,
+    SparkForgeStepSpec,
+    StepKind,
+    LayerKind,
+    adapt_pipeline,
+    debug_request_from_sparkforge,
+    enrich_plan,
+)
+from etlantic.plan import plan_pipeline
+
+spec = SparkForgePipelineSpec(
+    name="ecommerce",
+    schema="demo",
+    steps=(
+        SparkForgeStepSpec(
+            name="orders",
+            kind=StepKind.BRONZE_RULES,
+            layer=LayerKind.BRONZE,
+            table_name="bronze_orders",
+        ),
+        SparkForgeStepSpec(
+            name="clean_orders",
+            kind=StepKind.SILVER_TRANSFORM,
+            layer=LayerKind.SILVER,
+            source="orders",
+            table_name="silver_orders",
+            write_mode="overwrite",
+        ),
+    ),
+)
+adapted = adapt_pipeline(spec)
+adapted.pipeline_cls.validate(profile=adapted.profile)
+plan = enrich_plan(
+    plan_pipeline(adapted.pipeline_cls, profile=adapted.profile),
+    adapted,
+)
+request = debug_request_from_sparkforge(mode="incremental", skip_writes=True)
+```
+
+## Delivery direction
+
+1. **Plan-only** — generate/inspect ETLantic plans from SparkForge IR
+   (`strict_delta=False` to warn instead of fail when Delta caps are unknown)
+2. **Dual reporting** — `adapt_run_result` → `PipelineRunReport`
+3. **ETLantic planning** — selections/intents via `debug_request_from_sparkforge`
+4. **Plugin execution** — `Profile.spark_engine="pyspark"` / SQL plugins
+5. **Native facade** — Medallantic owns medallion authoring and retires the
+   duplicated legacy engine-specific builders
+
+`transform_ref` / bronze `rules` emit `PMSF411` warnings: the adapter builds
+passthrough transforms for planning parity; it does not execute SparkForge
+callables. Write intents (including MERGE) are attached via `enrich_plan` for
+orchestration; the local runtime still gates materialization with
+`RunRequest.no_write`.
+
+See `docs/11_DEVELOPMENT/MIGRATION_0_9_TO_0_10.md`.
+
+## Boundary
+
+| Concern | Owner |
+|---|---|
+| bronze / silver / gold APIs | Medallantic |
+| portable graph, plan, reports | ETLantic |
+| legacy mapping + parity fixtures | Medallantic migration layer |
+| physical execution | ETLantic engine and storage plugins |

medallantic-0.27.0/ROADMAP.md

@@ -0,0 +1,372 @@
+# Medallantic Roadmap
+
+Medallantic is the engine-agnostic medallion pipeline facade built on
+[ETLantic](https://github.com/eddiethedean/etlantic). It will provide one
+bronze/silver/gold authoring model across local Python, Polars, Pandas, SQL,
+and PySpark while delegating contracts, graph validation, deterministic
+planning, execution coordination, reports, and plugin trust to ETLantic.
+
+This roadmap defines “full parity” against both legacy SparkForge surfaces:
+
+- `pipeline_builder` (PySpark/Sparkless)
+- `sql_pipeline_builder` (SQLAlchemy/Moltres)
+
+Parity means preserving supported user outcomes and semantics, not copying
+the two legacy implementations or exposing backend objects in the portable
+model.
+
+## Product boundary
+
+Medallantic owns:
+
+- bronze, silver, and gold vocabulary and conventions
+- the fluent medallion builder and declarative configuration
+- layer-aware dependency defaults and quality policies
+- migration from both SparkForge builders
+- medallion-oriented run results, explanations, and documentation
+
+ETLantic owns:
+
+- typed pipeline contracts and the logical graph
+- validation, diagnostics, plans, run requests, and normalized reports
+- artifact, state, lineage, security, and plugin coordination
+- backend-neutral execution and compilation contracts
+
+Execution plugins own:
+
+- dataframe and relational execution
+- SQL dialect and database transaction behavior
+- Spark session and cluster behavior
+- Delta Lake and other storage-specific operations
+- engine-native expression compilation and optimization
+
+Medallantic will not recreate separate Spark and SQL builder hierarchies.
+Unsupported behavior must fail with a capability diagnostic; it must never
+silently change semantics or select another engine.
+
+### Promoting capabilities into ETLantic
+
+ETLantic may be extended whenever Medallantic reveals a capability that is
+useful beyond medallion architecture. Promotion is expected for portable
+concepts such as typed multi-output steps, state transitions, materialization
+policies, transaction boundaries, plan diagnostics, or plugin capability
+negotiation.
+
+A capability belongs in ETLantic when it:
+
+- has the same semantics outside bronze/silver/gold pipelines,
+- must be shared by more than one facade or execution plugin,
+- affects portable validation, planning, execution, reports, or security, and
+- can be named without medallion or engine-specific terminology.
+
+Layer names, layer defaults, medallion dependency conventions, and migration
+compatibility remain in Medallantic. Each milestone may therefore include
+prerequisite Etlantic work; those prerequisites should land with their own
+core tests before Medallantic consumes them.
+
+## Current baseline
+
+The renamed package starts with the former `etlantic-sparkforge` IR adapter.
+It currently provides:
+
+- SparkForge medallion IR parsing
+- bronze/silver/gold-to-ETLantic graph adaptation
+- dependency validation and cycle rejection
+- layer quality-threshold mapping
+- write-intent and retry-policy mapping
+- run-intent and run-selection mapping
+- plan enrichment
+- SparkForge result normalization with secret redaction
+- Delta capability checks
+- IR fixture parity tests without SparkForge or PySpark installed
+
+The baseline does **not** yet provide a native `MedallionPipeline` builder,
+execute legacy transformation callables, or reach live Spark/SQL behavioral
+parity.
+
+## Joint pre-1.0 sequence
+
+Medallantic milestones are release-gated with ETLantic's pre-1.0 roadmap:
+
+| ETLantic release | Medallantic phase | Joint focus |
+|---|---|---|
+| 0.27 | M0 (partial) | Rename, workspace/CI, first `medallantic` PyPI publish |
+| 0.28 | M0 closeout | Quadruple-minor burn-in, Plugin `/1` freeze, facade boundary |
+| 0.29 | M1 | Native medallion authoring and facade conformance |
+| 0.30 | M2 | Portable quality/rule semantics |
+| 0.31 | M3 | Execution, state, writes, and materialization |
+| 0.32 | M4 | PySpark/SparkForge differential parity |
+| 0.33 | M5 | SQLAlchemy/relational differential parity |
+| 0.34 | M6 | Operations, evidence, and production readiness |
+| 0.35 | M7 | Migration completion and joint freeze |
+| 0.36–0.98 | — | Joint compatibility burn-in |
+| 0.99 | — | Shared release-candidate rehearsal |
+
+The [ETLantic roadmap](../../ROADMAP.md) is authoritative for the core
+capability promoted at each phase and the joint exit gate. This document is
+authoritative for Medallantic feature parity and migration scope.
+
+## Definition of full parity
+
+Full parity is reached only when all required rows below have:
+
+1. a public Medallantic API,
+2. an explicit ETLantic mapping,
+3. conformance tests shared by all applicable engines,
+4. live PySpark and SQL integration tests where backend behavior matters,
+5. migration documentation and a declared compatibility result.
+
+“Equivalent” permits an intentional API improvement. “Plugin” means the
+portable intent is in Medallantic/ETLantic and the physical behavior is
+implemented by a capability-bearing plugin. “Do not carry forward” records
+legacy behavior that would be unsafe or architecturally incorrect.
+
+## Parity matrix
+
+| Capability | Spark builder | SQL builder | Target | Priority |
+|---|---:|---:|---|---|
+| Fluent bronze/silver/gold builder | Yes | Yes | Native `MedallionPipeline`/builder | P0 |
+| Partial pipelines (any layer subset) | Yes | Yes | Equivalent | P0 |
+| Unique step names and construction-time validation | Yes | Yes | Equivalent diagnostics | P0 |
+| Automatic dependency ordering | Yes | Yes | ETLantic deterministic planner | P0 |
+| Cycle and missing-dependency detection | Yes | Yes | Fail closed with stable codes | P0 |
+| Bronze source to silver wiring | Yes | Yes | Typed output references | P0 |
+| Multiple silver inputs for gold | Yes | Yes | Named typed input ports | P0 |
+| Prior silver result access | Yes | Yes | Run-scoped artifact context | P0 |
+| Cross-schema reads and writes | Yes | Yes | Dataset bindings plus profiles | P0 |
+| Layer descriptions and metadata | Yes | Partial | Portable metadata | P1 |
+| Per-layer validation thresholds | Yes | Yes | Named medallion quality policy | P0 |
+| Multiple rules per column | Yes | Yes | Contract-backed quality gates | P0 |
+| Valid/invalid row separation | Yes | Yes | Typed accepted/rejected outputs | P0 |
+| Validation-only steps and runs | Yes | Yes | `RunIntent.VALIDATE`, no writes | P0 |
+| PySpark expression rules | Yes | No | PySpark compiler/plugin | P0 |
+| String rule shorthand | Yes | No | Medallantic rule DSL | P1 |
+| Moltres expression rules | No | Yes | SQL compiler compatibility | P0 |
+| SQLAlchemy expression compatibility | No | Yes | SQL migration adapter | P0 |
+| Callable transforms | Yes | Yes | Backend implementation refs | P0 |
+| Initial load | Yes | Yes | Initialize intent | P0 |
+| Incremental load | Yes | Yes | Incremental strategy plus state | P0 |
+| Full refresh | Yes | Equivalent | Refresh intent | P0 |
+| Bronze incremental column | Yes | Yes | Watermark strategy | P0 |
+| Silver watermark column | Yes | Yes | State strategy | P0 |
+| Append writes | Yes | Yes | Portable write intent | P0 |
+| Overwrite writes | Yes | Yes | Portable write intent | P0 |
+| Partition overwrite | Yes | Backend-specific | Plugin capability | P1 |
+| Merge/upsert | Delta | Database-specific | Keyed merge capability | P0 |
+| Ignore/skip-if-exists | Yes | Partial | Portable write intent | P1 |
+| Schema compatibility checks | Yes | ORM model | Contracts plus sink checks | P0 |
+| Schema override/evolution policy | Yes | ORM model | Explicit mutation policy | P1 |
+| Table create/drop/refresh | Yes | Yes | Storage plugin | P0 |
+| Silver initial recreate | N/A/overwrite | Yes | Refresh policy | P0 |
+| Gold replace on every run | Yes | Yes | Layer default, overridable | P0 |
+| SQLAlchemy ORM model table creation | No | Yes | SQL plugin compatibility | P0 |
+| Primary-key validation | No | Yes | Merge/publication validation | P0 |
+| Moltres lazy DataFrame transforms | No | Yes | SQL compiler | P0 |
+| SQLAlchemy `Select`/compound selects | No | Yes | SQL compiler | P0 |
+| Sync SQLAlchemy sessions | No | Yes | SQL runtime | P0 |
+| Async SQLAlchemy sessions | No | Yes | SQL runtime | P1 |
+| Multi-database SQLAlchemy support | No | Yes | Dialect conformance matrix | P1 |
+| Spark and Sparkless execution | Yes | No | PySpark plus local test backend | P0 |
+| JDBC sources and targets | Yes | No | SQL/PySpark I/O plugins | P1 |
+| SQLAlchemy sources and targets from Spark | Yes | No | Explicit bridge capability | P2 |
+| Delta merge | Yes | No | Delta plugin | P0 |
+| Delta optimize, vacuum, history | Yes | No | Delta plugin | P1 |
+| Delta time travel | Yes | No | Delta source capability | P1 |
+| Run one / run until | Yes | No | ETLantic run selection | P0 |
+| Stateful debug session | Yes | No | Medallantic convenience facade | P1 |
+| Runtime parameter overrides | Yes | No | ETLantic run request | P1 |
+| Transform/rule overrides for debug | Yes | No | Scoped implementation overrides | P1 |
+| Downstream invalidation and rerun | Yes | No | Artifact provenance/invalidation | P1 |
+| Skip writes during debugging | Yes | No | No-write materialization policy | P0 |
+| Retry attempts and delays | Yes | Shared base | ETLantic retry policy | P0 |
+| Contextual pipeline/step logging | Yes | Yes | Structured lifecycle events | P0 |
+| Normalized run and step reports | Yes | Yes | Medallantic result facade | P0 |
+| Counts, duration, validation and table metrics | Yes | Yes | Normalized report fields | P0 |
+| Persisted execution-log writer | Yes | Shared base | Observability provider | P1 |
+| Log table create/append/read | Yes | Shared base | Provider conformance | P1 |
+| Trend and quality analytics | Yes | No | Event consumer/reference provider | P2 |
+| Anomaly detection queries | Yes | No | Observability integration | P2 |
+| Parallel candidates/execution groups | Analysis | Async SQL claim | Planner explain plus executor | P1 |
+| Dependency recommendations | Yes | Shared base | Structured lint findings | P2 |
+| Development/testing/production presets | Yes | Shared base | Profile templates | P1 |
+| Serialization of configuration/results | Yes | Yes | Stable versioned schemas | P0 |
+| Detailed errors with suggestions | Yes | Yes | Stable diagnostics/remediation | P0 |
+| Engine-explicit configuration | Partial | Partial | Required profile selection | P0 |
+| Silent runtime/mock auto-detection | Yes | N/A | Do not carry forward | Never |
+| Automatic cycle breaking | Legacy analyzer | Legacy analyzer | Do not carry forward | Never |
+| Separate public builders per engine | Yes | Yes | Do not carry forward | Never |
+
+## Delivery milestones
+
+### M0 / ETLantic 0.27–0.28 — Rename, release hygiene, and M0 closeout
+
+**Shipped in 0.27.0**
+
+- [x] Rename distribution to `medallantic`.
+- [x] Rename import package to `medallantic`.
+- [x] Rename workspace paths, tests, documentation, extras, and release checks.
+- [x] Preserve SparkForge-specific conversion helper names where they describe
+  the legacy input format.
+- [x] Publish the first `medallantic` distribution (`medallantic==0.27.0`).
+
+**Owned by 0.28 (M0 closeout)**
+
+- [ ] Decide whether to publish a final `etlantic-sparkforge` compatibility
+  release that depends on Medallantic and emits a deprecation warning.
+- [ ] Document facade-package release category and ongoing trusted-publishing
+  evidence for `medallantic`.
+
+Exit criteria: clean build, install, import, docs, lockfile, and adapter suite on
+every release; core remains importable without Medallantic; no medallion
+identifier in ETLantic wire schemas.
+
+### M1 / ETLantic 0.29 — Native medallion authoring
+
+- [ ] Add `MedallionPipeline`, `MedallionBuilder`, `Bronze`, `Silver`, and
+  `Gold` public surfaces.
+- [ ] Support fluent and declarative/serialized authoring.
+- [ ] Map every layer definition to ordinary ETLantic nodes, typed ports,
+  quality gates, sinks, and policies.
+- [ ] Support partial pipelines, multiple branches, prior-result references,
+  cross-schema bindings, descriptions, tags, and deterministic names.
+- [ ] Add stable `MDL1xx` construction and graph diagnostics.
+- [ ] Keep the current IR adapter as `medallantic.migrate.sparkforge`.
+
+Exit criteria: representative SparkForge pipelines can be authored natively,
+validated, planned, serialized, and explained without SparkForge installed.
+
+### M2 / ETLantic 0.30 — Quality and rules parity
+
+- [ ] Define an engine-neutral rule AST/DSL for common shorthand:
+  `not_null`, comparisons, membership, ranges, regex, length, uniqueness,
+  and custom contract rules.
+- [ ] Compile rules to ETLantic/ContractModel quality gates.
+- [ ] Add PySpark, SQL/Moltres, Polars, and Pandas rule compilers.
+- [ ] Return accepted and rejected typed artifacts with counts and reasons.
+- [ ] Implement named per-layer defaults and per-step overrides.
+- [ ] Make validation cost and unsupported-rule fallback visible in plans.
+
+Exit criteria: shared fixtures produce equivalent pass/fail decisions,
+accepted/rejected counts, and diagnostics across supported engines.
+
+### M3 / ETLantic 0.31 — Execution and materialization parity
+
+- [ ] Execute native callable transforms through ETLantic implementation
+  references.
+- [ ] Implement initial, incremental, refresh, standard, and validation-only
+  behavior.
+- [ ] Map incremental columns and watermarks to atomic ETLantic state
+  transitions.
+- [ ] Implement append, replace, keyed merge, skip-if-exists, and partition
+  replacement with capability checks.
+- [ ] Implement layer defaults for bronze preservation, silver refresh, and
+  gold publication without hard-coding them into ETLantic core.
+- [ ] Support retries only when the planned operation is safely retryable.
+- [ ] Normalize run, step, validation, artifact, write, and state results.
+
+Exit criteria: engine-neutral conformance fixtures pass for local, Polars,
+Pandas, SQL, and PySpark where each engine advertises support.
+
+### M4 / ETLantic 0.32 — PySpark/SparkForge parity
+
+- [ ] Add a live migration bridge for `PipelineBuilder` definitions.
+- [ ] Support PySpark Column validation expressions and transform callables.
+- [ ] Support real PySpark and Sparkless test modes explicitly.
+- [ ] Cover schema/catalog management, cross-schema access, caches, JDBC I/O,
+  and Spark-native metrics through plugins.
+- [ ] Add Delta capabilities for merge, optimize, vacuum, history, schema
+  evolution, and time travel.
+- [ ] Reproduce run-one, run-until, no-write, parameter override, transform
+  override, rerun, and downstream invalidation workflows.
+- [ ] Compare logical order, writes, validation outcomes, and normalized
+  reports against frozen SparkForge fixtures.
+
+Exit criteria: every supported `pipeline_builder` fixture has a documented
+`equivalent`, `plugin-dependent`, or `intentionally rejected` result.
+
+### M5 / ETLantic 0.33 — SQL pipeline-builder parity
+
+- [ ] Add a live migration bridge for `SqlPipelineBuilder` definitions.
+- [ ] Accept Moltres DataFrames/expressions and SQLAlchemy ORM, `Select`, and
+  compound-select sources during migration.
+- [ ] Preserve lazy relational execution and avoid unnecessary table
+  round-trips between steps.
+- [ ] Support model-driven table creation and primary-key validation.
+- [ ] Match initial/incremental silver behavior and gold replacement defaults.
+- [ ] Support sync and async SQLAlchemy execution with transaction rollback.
+- [ ] Validate missing sources, ambiguous/duplicate columns, invalid transform
+  results, conversion failures, and write failures consistently.
+- [ ] Establish dialect tiers and live CI for SQLite and PostgreSQL, followed
+  by additional SQLAlchemy dialects.
+
+Exit criteria: the SQL parity matrix passes on SQLite and PostgreSQL, with
+other dialect support accurately capability-gated.
+
+### M6 / ETLantic 0.34 — Operations, observability, and production readiness
+
+- [ ] Provide medallion-oriented structured explain output.
+- [ ] Emit normalized lifecycle events with pipeline, layer, step, attempt,
+  plan, run, and backend context.
+- [ ] Add durable run-history provider conformance for create/append/read.
+- [ ] Port trend, quality, performance, and anomaly analytics as optional
+  event consumers rather than core storage logic.
+- [ ] Add development, testing, and production profile templates.
+- [ ] Complete redaction, plugin allowlist, schema-mutation, concurrency,
+  cancellation, timeout, and recovery tests.
+- [ ] Publish compatibility, security, performance, and production-readiness
+  documentation with explicit support tiers.
+
+Exit criteria: Medallantic meets ETLantic’s production profile requirements
+and has no undocumented fallback or mutation behavior.
+
+### M7 / ETLantic 0.35 — Migration completion
+
+- [ ] Ship an automated scanner that inventories a SparkForge project and
+  emits a migration report.
+- [ ] Generate native Medallantic definitions where conversion is safe.
+- [ ] Produce stable diagnostics for manual conversion points.
+- [ ] Maintain golden before/after plans and run reports.
+- [ ] Publish versioned deprecation timelines for legacy imports.
+- [ ] Remove transitional adapters only in a major release.
+
+Exit criteria: both legacy builders have documented, tested migration paths
+and all claimed parity rows are backed by conformance evidence.
+
+## Test strategy
+
+The parity suite will have four layers:
+
+1. **IR tests** — no SparkForge, SQLAlchemy, or PySpark dependency.
+2. **Semantic conformance** — one logical fixture and normalized assertions
+   across local, Polars, Pandas, SQL, and PySpark.
+3. **Legacy differential tests** — run frozen SparkForge and Medallantic
+   fixtures and compare graph order, validation, writes, and reports.
+4. **Backend integration tests** — real SQLite/PostgreSQL, PySpark, and Delta
+   environments for physical semantics.
+
+Every parity claim must identify its fixture, engines, capability requirements,
+and normalized assertions. Unit-test coverage alone cannot establish backend
+parity.
+
+## Compatibility policy
+
+- SparkForge names may remain in migration-only APIs and serialized legacy IR.
+- New authoring APIs use Medallantic names exclusively.
+- Diagnostic codes and serialized fields already consumed externally remain
+  stable until a documented major-version migration.
+- Backend-specific expressions are accepted only as implementation details;
+  portable definitions use contracts, rule AST nodes, and typed references.
+- A capability gap is an explicit diagnostic, never an implicit fallback.
+
+## Release gates
+
+A release may claim full parity only when:
+
+- every P0 matrix row is complete,
+- all P1 rows are complete or explicitly deferred with a documented support
+  limitation,
+- both legacy differential suites pass,
+- SQLite, PostgreSQL, PySpark, and Delta integration suites pass,
+- package build/install/import and documentation checks pass,
+- security/redaction and schema-mutation gates pass,
+- the compatibility table is generated from test evidence.