floe-python 0.8.3__tar.gz → 0.8.4__tar.gz

{floe_python-0.8.3 → floe_python-0.8.4}/CHANGELOG.md

@@ -2,6 +2,30 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.4
+
+- **Columns that normalise to the same name now fail `floe validate`, not the run (#1005).** With
+  `normalize_columns`, a column `order_id` and a JSON or XML selector `order.id` (or `order/id`) both become
+  `order_id`; `floe validate` passed and `floe run` then failed. Validate now rejects it, and the message names both
+  columns or selectors and the shared name.
+
+- **`add_columns` schema evolution on Iceberg is reported and emitted.** An entity run that adds columns to
+  the table now sets `schema_evolution.applied` and lists them in `schema_evolution.added_columns` in `run.json`,
+  emits the `schema_evolution_applied` event, and `schema_evolution.enabled` is `true` whenever the mode is
+  `add_columns`. They were always `false` and the event was never emitted. (#994)
+
+- **Accepted rows now reach the sink chunk by chunk (#1056)**: an entity no longer holds all its accepted rows
+  in memory before the write. The Parquet sink writes parts as the input is read, and the Iceberg sink writes
+  data files into the same single commit. A file's chunks after the first wait on local disk until the file
+  is read to the end, so a late read error or an `abort` still drops all of that file's rows. `append`
+  stages its Parquet parts like `overwrite` does, so a killed run leaves no visible part for a retry to duplicate.
+
+- **`unique` checks with `append` read the target's keys once per entity (#1057)**: an Iceberg table's key columns
+  were scanned once per chunk, and a Parquet sink's parts were loaded into memory in full, so cost and memory
+  grew with chunks times history. The key columns are now written once to a local file in the entity's temp
+  directory, and each chunk joins its keys against that file. Flags and counts are unchanged; the temp
+  directory needs room for the history's key columns.
+
 ## v0.8.3
 
 - **`-p env:NAME` (#1068).** `-p` / `--profile` and `profile_path=` accept `env:NAME`, which reads the profile YAML
@@ -19,7 +43,21 @@ All notable changes to Floe are documented in this file.
 
 ## Unreleased
 
-### dagster-floe
+### airflow-floe
+
+- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
+  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
+
+## dagster-floe v0.4.2
+
+- **Silver lineage events from a `kubernetes_job` run now carry the Dagster parent run (#1076).** The Job
+  container gets `DAGSTER_RUN_ID` and `DAGSTER_JOB_NAME`, and the Job and its pod a `dagster/run-id` label;
+  both `job_scope`s were missing them.
+
+## dagster-floe v0.4.1
+
+**Upgrading from 0.4.0:** the `floe` extra now requires Floe 0.8.3 or newer, and so does the image a
+`kubernetes_job` runs when the run has a profile.
 
 - **A `kubernetes_job` Job gets the profile as content, not as a pod-local path (#1069).** A run with a
   profile reads it in the run pod and sets it as the Job's `FLOE_PROFILE_CONTENT` env var, with
@@ -31,11 +69,6 @@ All notable changes to Floe are documented in this file.
   writing tables, so a re-run could collide with it. A Job that reached a final state is left alone, so
   `ttl_seconds_after_finished` still controls how long its logs stay.
 
-### airflow-floe
-
-- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
-  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
-
 ## dagster-floe v0.4.0
 
 **Upgrading from 0.3:** rejected assets move to new keys (below). Works with Floe 0.8.1 or newer; a Glue

{floe_python-0.8.3 → floe_python-0.8.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.3
+Version: 0.8.4
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

floe_python-0.8.4/conformance/config/val2_normalize_selector_collision/config.yml

@@ -0,0 +1,21 @@
+version: '0.1'
+entities:
+- name: orders
+  source:
+    format: json
+    path: in/orders.json
+  sink:
+    accepted:
+      format: parquet
+      path: out/accepted
+  policy:
+    severity: warn
+  schema:
+    normalize_columns:
+      enabled: true
+    columns:
+    - name: order_id
+      type: string
+    - name: order_ref
+      type: string
+      source: order.id

floe_python-0.8.4/conformance/config/val2_normalize_selector_collision/expected/validate.json

@@ -0,0 +1,4 @@
+{
+  "ok": false,
+  "error": "entity.name=orders column source collision: order_id and order.id -> order_id"
+}

{floe_python-0.8.3 → floe_python-0.8.4}/context/architecture.md

@@ -59,7 +59,7 @@ The CLI and the Python API share the same functions: `_cli.py` parses arguments
 |---|---|
 | Polars | Reading (CSV, JSON, Parquet, Avro), casting, checks, the accepted/rejected split, Parquet and CSV output |
 | PyIceberg (+ `pyiceberg-core`) | Polaris REST catalog, table creation, schema evolution, overwrite/append commits, maintenance |
-| DuckDB (pinned) | Iceberg `MERGE INTO`, unique checks against table history, compaction; spills to disk |
+| DuckDB (pinned) | Iceberg `MERGE INTO`, compaction; spills to disk |
 | pyarrow | Arrow exchange between the three, ORC reads |
 | fastexcel | XLSX |
 | boto3 | S3 storage client |
@@ -80,8 +80,8 @@ them (see `context/decisions/polars-and-arrow.md`).
    → reject_file: copy the raw file to the rejected location, next file
 
 3. Row-level checks (per file, in bounded chunks)
-   cast to declared types → not_null → unique (within the run; with `append`, seeded from
-   the accepted sink) → rows with errors get `__floe_row_index` + `__floe_errors`
+   cast to declared types → not_null → unique (within the run; with `append`, against the
+   accepted sink's key columns, cached once per entity in the entity's temp dir and joined per chunk) → rows with errors get `__floe_row_index` + `__floe_errors`
    → severity decides: warn keeps them, reject routes them to rejected CSV, abort fails the entity
 
 4. Write outputs (once per entity)

{floe_python-0.8.3 → floe_python-0.8.4}/docs/config.md

@@ -522,7 +522,8 @@ See `docs/pii.md` for full strategy reference and masking pattern syntax.
   - `enabled`: boolean toggle.
   - `strategy`: `snake_case`, `lower`, `camel_case`, `none`.
   - When enabled, both schema column names and input column names are normalized
-    before checks. If normalization causes a name collision, the run fails.
+    before checks. Two schema columns (or selectors) that normalise to the same name fail `floe validate`; an input column
+    that collides with another after normalization fails the run.
 - `schema_evolution` (optional, v0.2)
   - Omitted by default, Floe behaves as `mode: strict` and `on_incompatible: fail`.
   - `mode`: `strict` or `add_columns`.

{floe_python-0.8.3 → floe_python-0.8.4}/docs/how-it-works.md

@@ -7,9 +7,11 @@ happened in machine-readable reports.
 
 The runtime is a lightweight single-node Python package. Floe reads and validates
 with Polars, writes Iceberg tables through PyIceberg (one catalog commit per
-entity), and runs Iceberg merges and unique checks against the table's history
-through DuckDB, which spills to disk. The libraries exchange data as Arrow, so
-no extra serialization step sits between them.
+entity), and runs Iceberg merges through DuckDB, which spills to disk. With
+`append`, a unique check against the target's history reads the target's key
+columns once per entity into a local file, and each chunk checks its keys
+against that file. The libraries exchange data as Arrow, so no extra
+serialization step sits between them.
 
 This document describes the canonical ingestion pipeline Floe uses for each
 entity. The order is deterministic and is reflected in reports.

{floe_python-0.8.3 → floe_python-0.8.4}/docs/logging.md

@@ -28,13 +28,11 @@ The final line on stdout is always the `run_finished` event.
 - `entity_started`: entity name (`name`, `ts_ms`)
 - `file_started`: entity/input pair (`entity`, `input`, `ts_ms`)
 - `file_finished`: file outcome and counts (`status`, `rows`, `accepted`, `rejected`, `elapsed_ms`, `ts_ms`)
-- `schema_evolution_applied`: additive schema-evolution audit event (`entity`, `mode`, `added_columns`, `ts_ms`); reserved, not emitted in 0.8
+- `schema_evolution_applied`: additive schema-evolution audit event (`entity`, `mode`, `added_columns`, `ts_ms`); emitted once per entity run in which `add_columns` adds columns to an Iceberg table
 - `entity_finished`: entity totals (`status`, `files`, `rows`, `accepted`, `rejected`, `warnings`, `errors`, `ts_ms`)
 - `run_finished`: final run totals and exit code (`status`, `exit_code`, `files`, `rows`, `accepted`, `rejected`, `warnings`, `errors`, `summary_uri`, `ts_ms`)
 - `log`: structured warning/error line (`log_level`, `code`, `message`, optional `entity`, optional `input`, `ts_ms`)
 
-`schema_evolution_applied` is part of the event contract, but 0.8 does not emit it yet, even when `add_columns` adds columns to an Iceberg table.
-
 ### `--log-format off` (default)
 
 - Suppresses all structured event output.

{floe_python-0.8.3 → floe_python-0.8.4}/docs/python-bindings.md

@@ -137,7 +137,7 @@ The callback is **swappable between calls** — calling `set_observer` again rep
 | `run_finished` | `run_id`, `status`, `exit_code`, `files`, `rows`, `accepted`, `rejected`, `summary_uri` |
 | `log` | `run_id`, `log_level`, `code`, `message`, `entity`, `input` |
 
-All events have an `event` key (string discriminator) and `ts_ms` (Unix timestamp in milliseconds). Each event type is exported as a TypedDict (`floe.RunStartedEvent`, …, `floe.LogEvent`; `floe.RunEvent` is their union), defined in [`src/floe/run/events.py`](../src/floe/run/events.py). `schema_evolution_applied` is part of the contract but is not emitted in 0.8.
+All events have an `event` key (string discriminator) and `ts_ms` (Unix timestamp in milliseconds). Each event type is exported as a TypedDict (`floe.RunStartedEvent`, …, `floe.LogEvent`; `floe.RunEvent` is their union), defined in [`src/floe/run/events.py`](../src/floe/run/events.py).
 
 ---
 

{floe_python-0.8.3 → floe_python-0.8.4}/docs/report.md

@@ -106,11 +106,13 @@ Merge-specific metrics (Iceberg `merge_scd1`):
 
 ### `schema_evolution` (entity report)
 
-- `enabled`, `applied`, `incompatible_changes_detected`: always `false` in 0.8
+- `enabled`: `true` when `mode` is `add_columns`
 - `mode`: resolved config mode (`strict` or `add_columns`)
+- `applied`: `true` when Floe added columns to the Iceberg table during the write (D13)
+- `added_columns`: ordered list of the columns added in that write; present only when `applied` is `true`
+- `incompatible_changes_detected`: always `false` in 0.8 (a non-additive change fails the entity before the report)
 
-`mode: add_columns` does add new columns to an Iceberg table (D13), but 0.8 does not record
-them here and emits no `schema_evolution_applied` event yet.
+Floe also emits a `schema_evolution_applied` event when it adds columns (see [Logging](logging.md)).
 
 Notes:
 - Metrics are populated when the writer can collect them cheaply and reliably.

{floe_python-0.8.3 → floe_python-0.8.4}/docs/sinks/parquet.md

@@ -26,7 +26,10 @@ Semantics:
 
 Large accepted datasets are split into multiple part files. The split threshold is controlled by
 `sink.accepted.options.max_size_per_file` (default 256 MB), measured on the in-memory size of
-the rows. The entity's accepted rows are written once, after every input file is read.
+the rows. Accepted rows are written as the input is read, so memory holds about one part plus one
+chunk of input, not the whole entity. Every part is staged as a `_staging-<id>-part-*` file and
+copied to its `part-*` name only once the whole input was read and written, so a write that fails
+(or a killed run) leaves no visible part of that run; orphaned `_staging-*` files are safe to delete.
 
 ```yaml
 sink:

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/README.md

@@ -30,18 +30,20 @@ Current model:
 ## Install
 
 Prereqs:
-- Floe 0.8.1 or newer (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
+- Floe 0.8.3 or newer (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
 - Python 3.10+.
 
 ```bash
 pip install dagster-floe
-pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.1,<0.9 for the in-process runner
+pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.3,<0.9 for the in-process runner
 ```
 
 ## Supported Floe versions
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.2 | 0.8.3 or newer | `0.8` |
+| 0.4.1 | 0.8.3 or newer (`kubernetes_job` runs pass the profile as `-p env:FLOE_PROFILE_CONTENT`) | `0.8` |
 | 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |
 | 0.3.1 | 0.8.1 or newer (`job_scope: domain` Jobs run `floe run --domain`) | `0.8` |
 | 0.3.0 | 0.8 | `0.8` |
@@ -164,7 +166,7 @@ From 0.3.0, remove that patch and set the variables above.
 
 ## Lineage integration
 
-The connector automatically injects two environment variables into every floe subprocess:
+The connector automatically injects two environment variables into every floe run it starts locally or as a `kubernetes_job` (not `databricks_job`):
 
 - `DAGSTER_RUN_ID` — the current Dagster run ID
 - `DAGSTER_JOB_NAME` — the Dagster job name

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.4.0"
+version = "0.4.2"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"
@@ -33,7 +33,7 @@ kubernetes = [
   "kubernetes>=28",
 ]
 floe = [
-  "floe-python>=0.8.1,<0.9",
+  "floe-python>=0.8.3,<0.9",
 ]
 remote = [
   "fsspec",

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -561,6 +561,7 @@ def _make_domain_multi_asset(
         run_id = getattr(run, "run_id", None) if run is not None else None
         # concurrent groups of one Dagster run must not share a Floe run id (it names the summary file)
         floe_run_id = f"{run_id}_{context.op_def.name}" if run_id else None
+        dagster_job_name = getattr(run, "job_name", None) if run is not None else None
         names = [entity.name for entity in chosen]
         # the whole domain of the manifest: `floe run --domain` says it; a custom template stays as written
         whole_domain = bool(names) and set(names) == domain_names and execution.per_entity_args == [
@@ -575,6 +576,8 @@ def _make_domain_multi_asset(
             entities=names,
             execution=replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution,
             runner_definition=runner_definition,
+            dagster_run_id=run_id,
+            dagster_job_name=dagster_job_name,
         )
         finished = _parse_finished(context, result, execution, run_id)
         summary = _load_run_summary(context, finished, config_uri)

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/kubernetes_runner.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import logging
+import re
 import time
 import uuid
 from dataclasses import replace
@@ -25,6 +26,8 @@ _log = logging.getLogger(__name__)
 _PROFILE_ENV = "FLOE_PROFILE_CONTENT"
 _DEFAULT_POLL_INTERVAL_S = 10
 _DEFAULT_TIMEOUT_S = 3600
+_RUN_ID_LABEL = "dagster/run-id"
+_LABEL_VALUE = re.compile(r"[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?")
 
 
 def run_kubernetes_job(
@@ -34,13 +37,17 @@ def run_kubernetes_job(
     runner: ManifestRunnerDefinition,
     jobs_api: Any = None,
     core_api: Any = None,
+    dagster_run_id: str | None = None,
+    dagster_job_name: str | None = None,
 ) -> RunResult:
     namespace = runner.namespace
     if not namespace:
         raise ValueError("kubernetes_job runner requires a non-empty 'namespace' field")
 
     job_name = _make_job_name(entity)
-    job_spec = build_k8s_job_spec(job_name, cmd_args, runner)
+    job_spec = build_k8s_job_spec(
+        job_name, cmd_args, runner, dagster_run_id=dagster_run_id, dagster_job_name=dagster_job_name
+    )
 
     poll_interval = runner.poll_interval_seconds or _DEFAULT_POLL_INTERVAL_S
     timeout = runner.timeout_seconds or _DEFAULT_TIMEOUT_S
@@ -112,7 +119,14 @@ def _delete_job(jobs_api: Any, job_name: str, namespace: str) -> None:
         _log.warning("could not delete Kubernetes Job %s/%s", namespace, job_name, exc_info=True)
 
 
-def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunnerDefinition) -> dict[str, Any]:
+def build_k8s_job_spec(
+    job_name: str,
+    cmd_args: list[str],
+    runner: ManifestRunnerDefinition,
+    *,
+    dagster_run_id: str | None = None,
+    dagster_job_name: str | None = None,
+) -> dict[str, Any]:
     if not runner.image:
         raise ValueError("kubernetes_job runner requires a non-empty 'image' field")
     if not runner.namespace:
@@ -144,6 +158,12 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
                 }
             )
 
+    # the lineage parent run (`floe.lineage.events.parent`): the Job has no Dagster process env
+    if dagster_run_id:
+        env_vars.append({"name": "DAGSTER_RUN_ID", "value": dagster_run_id})
+    if dagster_job_name:
+        env_vars.append({"name": "DAGSTER_JOB_NAME", "value": dagster_job_name})
+
     if profile_text is not None:
         env_vars.append({"name": _PROFILE_ENV, "value": profile_text})
 
@@ -176,8 +196,10 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
         pod_spec["tolerations"] = runner.tolerations
 
     # the defaults win: `floe-job` is how the runner finds the pod again
+    run_label = {_RUN_ID_LABEL: dagster_run_id} if dagster_run_id and _LABEL_VALUE.fullmatch(dagster_run_id) else {}
+    job_labels = {"app": "floe-runner", **run_label}
     pod_metadata: dict[str, Any] = {
-        "labels": {**(runner.labels or {}), "app": "floe-runner", "floe-job": job_name}
+        "labels": {**(runner.labels or {}), **job_labels, "floe-job": job_name}
     }
     if runner.annotations:
         pod_metadata["annotations"] = runner.annotations
@@ -197,7 +219,7 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     return {
         "apiVersion": "batch/v1",
         "kind": "Job",
-        "metadata": {"name": job_name, "namespace": runner.namespace, "labels": {"app": "floe-runner"}},
+        "metadata": {"name": job_name, "namespace": runner.namespace, "labels": job_labels},
         "spec": job_spec,
     }
 

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/runner.py

@@ -32,6 +32,8 @@ class Runner:
         execution: ManifestExecution,
         runner_definition: ManifestRunnerDefinition,
         manifest_uri: str | None = None,
+        dagster_run_id: str | None = None,
+        dagster_job_name: str | None = None,
     ) -> RunResult:
         """One run for several entities of `domain`, one after another (`job_scope: domain`)."""
         raise NotImplementedError
@@ -64,6 +66,8 @@ class LocalRunner(Runner):
         execution: ManifestExecution,
         runner_definition: ManifestRunnerDefinition,
         manifest_uri: str | None = None,
+        dagster_run_id: str | None = None,
+        dagster_job_name: str | None = None,
     ) -> RunResult:
         if runner_definition.runner_type != "kubernetes_job":
             raise NotImplementedError("job_scope: domain needs a kubernetes_job runner")
@@ -81,7 +85,11 @@ class LocalRunner(Runner):
         from .kubernetes_runner import run_kubernetes_job, sized_for_domain
 
         return run_kubernetes_job(
-            args, entity=domain, runner=sized_for_domain(runner_definition, domain)
+            args,
+            entity=domain,
+            runner=sized_for_domain(runner_definition, domain),
+            dagster_run_id=dagster_run_id,
+            dagster_job_name=dagster_job_name,
         )
 
     def run_floe_entity(
@@ -110,7 +118,13 @@ class LocalRunner(Runner):
             )
             from .kubernetes_runner import run_kubernetes_job
 
-            return run_kubernetes_job(args, entity=entity, runner=runner_definition)
+            return run_kubernetes_job(
+                args,
+                entity=entity,
+                runner=runner_definition,
+                dagster_run_id=run_id,
+                dagster_job_name=dagster_job_name,
+            )
 
         if runner_definition is not None and runner_definition.runner_type == "databricks_job":
             if execution is None:

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/tests/test_domain_scope.py

@@ -92,6 +92,16 @@ def test_domain_scope_runs_the_entities_in_one_job_with_a_materialization_each(t
     assert args[args.index("--run-id") + 1].startswith(f"{result.run_id}_floe_")  # one Floe run id per group
 
 
+def test_domain_scope_job_carries_the_dagster_run_for_lineage(tmp_path: Path) -> None:
+    result, jobs_api = _run(tmp_path, dict.fromkeys(NAMES, "success"))
+
+    body = jobs_api.create_namespaced_job.call_args.kwargs["body"]
+    env = {e["name"]: e.get("value") for e in body["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert env["DAGSTER_RUN_ID"] == result.run_id
+    assert env["DAGSTER_JOB_NAME"]
+    assert body["metadata"]["labels"]["dagster/run-id"] == result.run_id
+
+
 def test_domain_scope_subset_runs_only_the_selected_entities(tmp_path: Path) -> None:
     selection = [AssetKey(["sales", "orders"]), AssetKey(["sales", "refunds"])]
     result, jobs_api = _run(tmp_path, {"orders": "success", "refunds": "success"}, selection=selection)

{floe_python-0.8.3 → floe_python-0.8.4}/orchestrators/dagster-floe/tests/test_kubernetes_runner.py

@@ -290,6 +290,80 @@ def test_run_floe_entities_creates_one_job_named_for_the_domain() -> None:
     assert container["resources"] == {"requests": {"cpu": "2"}}
 
 
+def test_run_floe_entities_job_carries_the_dagster_run_for_lineage() -> None:
+    execution = load_manifest(Path(__file__).parent / "fixtures" / "manifest.json").execution
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = _job(
+        succeeded=1, conditions=[{"type": "Complete", "status": "True"}]
+    )
+    core_api.list_namespaced_pod.return_value.items = []
+    runner = replace(_runner(), labels={"dagster/run-id": "hijacked"})
+
+    with (
+        patch("floe_dagster.kubernetes_runner._make_batch_client", return_value=jobs_api),
+        patch("floe_dagster.kubernetes_runner._make_core_client", return_value=core_api),
+    ):
+        LocalRunner().run_floe_entities(
+            config_uri="cfg.yml",
+            run_id="run-1_floe_sales",
+            domain="sales",
+            entities=["orders"],
+            execution=execution,
+            runner_definition=runner,
+            dagster_run_id="run-1",
+            dagster_job_name="daily",
+        )
+
+    body = jobs_api.create_namespaced_job.call_args.kwargs["body"]
+    env = {e["name"]: e["value"] for e in body["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert env["DAGSTER_RUN_ID"] == "run-1"  # the Dagster run, not the per-group Floe run id
+    assert env["DAGSTER_JOB_NAME"] == "daily"
+    assert body["metadata"]["labels"]["dagster/run-id"] == "run-1"
+    assert body["spec"]["template"]["metadata"]["labels"]["dagster/run-id"] == "run-1"  # the default wins
+
+
+def test_run_floe_entity_job_carries_the_dagster_run_for_lineage() -> None:
+    execution = load_manifest(Path(__file__).parent / "fixtures" / "manifest.json").execution
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = _job(
+        succeeded=1, conditions=[{"type": "Complete", "status": "True"}]
+    )
+    core_api.list_namespaced_pod.return_value.items = []
+
+    with (
+        patch("floe_dagster.kubernetes_runner._make_batch_client", return_value=jobs_api),
+        patch("floe_dagster.kubernetes_runner._make_core_client", return_value=core_api),
+    ):
+        LocalRunner().run_floe_entity(
+            config_uri="cfg.yml",
+            run_id="run-1",
+            entity="orders",
+            execution=execution,
+            runner_definition=_runner(),
+            dagster_job_name="daily",
+        )
+
+    body = jobs_api.create_namespaced_job.call_args.kwargs["body"]
+    env = {e["name"]: e["value"] for e in body["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert (env["DAGSTER_RUN_ID"], env["DAGSTER_JOB_NAME"]) == ("run-1", "daily")
+    assert body["metadata"]["labels"]["dagster/run-id"] == "run-1"
+
+
+def test_build_k8s_job_spec_without_a_dagster_run_adds_no_variables_or_label() -> None:
+    spec = build_k8s_job_spec("floe-orders-abc", ["floe", "run"], _runner())
+    env_names = {e["name"] for e in spec["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert not {"DAGSTER_RUN_ID", "DAGSTER_JOB_NAME"} & env_names
+    assert "dagster/run-id" not in spec["metadata"]["labels"]
+    assert "dagster/run-id" not in spec["spec"]["template"]["metadata"]["labels"]
+
+
+def test_build_k8s_job_spec_omits_a_run_id_label_that_is_not_a_valid_label_value() -> None:
+    spec = build_k8s_job_spec("floe-orders-abc", ["floe", "run"], _runner(), dagster_run_id="x" * 64)
+    env_names = {e["name"] for e in spec["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert "DAGSTER_RUN_ID" in env_names
+    assert "dagster/run-id" not in spec["metadata"]["labels"]
+
+
 def _pod_options_runner() -> ManifestRunnerDefinition:
     return replace(
         _runner(),

{floe_python-0.8.3 → floe_python-0.8.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.3"
+version = "0.8.4"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.3 → floe_python-0.8.4}/src/floe/checks/normalize.py

@@ -76,7 +76,7 @@ def source_names(columns: Sequence[ColumnConfig], strategy: str | None) -> list[
         name = normalize_name(source, strategy)
         if name in seen:
             raise FloeConfigError(
-                f"column source collision: {seen[name]} and {column.name} -> {name}"
+                f"column source collision: {seen[name]} and {source} -> {name}"
             )
         seen[name] = source
     return list(seen)

{floe_python-0.8.3 → floe_python-0.8.4}/src/floe/checks/unique.py

@@ -4,12 +4,13 @@
 # pyright: reportUnknownVariableType=false, reportUnknownArgumentType=false
 
 import math
-from collections.abc import Sequence
+from collections.abc import Mapping, Sequence
 from dataclasses import dataclass, field
 from datetime import datetime
 from decimal import Decimal
 
 import polars as pl
+import polars.selectors as cs
 import pyarrow as pa
 
 from floe.checks.rows import RowCheck
@@ -245,3 +246,52 @@ class UniqueTracker:
             for row in top.iter_rows(named=True)
         ]
         return UniqueResult(c.columns, c.duplicates, samples, c.target)
+
+
+def seed_from_history(
+    tracker: UniqueTracker,
+    history: pl.LazyFrame,
+    batch: pl.DataFrame,
+    stored: Mapping[str, str],
+) -> None:
+    """Seed `tracker` with the keys of `history` that `batch` also holds; `stored` maps runtime to history names.
+
+    A history key the batch lacks is never hit, so the flags are Rust's while memory follows the
+    batch: `history` is streamed, key columns only. The join matches -0.0 to 0.0 and NaN to any
+    NaN; the tracker then compares float bits, so those extra keys are never hit.
+    A key with a column `history` lacks (`add_columns` adds it on write) is skipped: historical
+    rows hold NULL there, which never duplicates.
+    """
+    columns = set(history.collect_schema().names())
+    for key in tracker.keys:
+        names = [stored.get(c, c) for c in key]
+        if not columns.issuperset(names):
+            continue
+        # positional names (k0, b0, ...): no stored column name can collide with a temporary
+        ids = [f"k{i}" for i in range(len(key))]
+        distinct = (
+            # by_name: a column named like a pattern (`*`, `^...$`) is selected literally
+            batch.select(cs.by_name(c).alias(k) for c, k in zip(key, ids, strict=True))
+            .drop_nulls()
+            .unique()
+        )
+        if not distinct.height:  # all-NULL keys are never checked; nothing to look up
+            continue
+        # distinct by float bits, as the tracker keys them: `unique` on a float merges -0.0 with 0.0
+        floats = [k for c, k in zip(key, ids, strict=True) if batch.schema[c].is_float()]  # fmt: skip
+        bits = {f"b{k}": pl.col(k).map_batches(_float_bits, return_dtype=pl.UInt64, is_elementwise=True) for k in floats}  # fmt: skip
+        subset = [k for k in ids if k not in floats] + list(bits)
+        scan = history.select(cs.by_name(n).alias(k) for n, k in zip(names, ids, strict=True))  # fmt: skip
+        # the history's dtypes can differ (unsigned widened, datetimes in us); a key its type
+        # cannot hold casts to NULL, which never matches
+        lookup = distinct.cast(scan.collect_schema(), strict=False).lazy()
+        hits = (
+            scan.join(lookup, on=ids, how="semi")
+            .with_columns(**bits)
+            # distinct keys only: memory follows the batch, not the history
+            .unique(subset)
+            .drop(*bits)
+            .collect(engine="streaming")
+        )
+        back = pl.Schema({k: batch.schema[c] for c, k in zip(key, ids, strict=True)})
+        tracker.seed(hits.cast(back).rename(dict(zip(ids, key, strict=True))))

{floe_python-0.8.3 → floe_python-0.8.4}/src/floe/config/validate.py

@@ -4,7 +4,12 @@ import re
 
 from floe import __version__
 from floe.checks.cast import decimal_type
-from floe.checks.normalize import normalize_name, output_names
+from floe.checks.normalize import (
+    normalize_name,
+    output_names,
+    resolve_strategy,
+    source_names,
+)
 from floe.config.project import (
     LineageConfig,
     RootConfig,
@@ -410,6 +415,10 @@ def validate_schema(entity: EntityConfig, version: str) -> None:
             f"schema.normalize_columns.strategy={strategy} is unsupported "
             f"(allowed: {', '.join(NORMALIZE_STRATEGIES)})",
         )
+    try:
+        source_names(schema.columns, resolve_strategy(schema))
+    except FloeConfigError as err:
+        raise _fail(entity, str(err)) from err
     mismatch = schema.mismatch
     if mismatch is not None:
         for field, policy, allowed in (