floe-python 0.8.2__tar.gz → 0.8.4__tar.gz

{floe_python-0.8.2 → floe_python-0.8.4}/CHANGELOG.md

@@ -2,28 +2,91 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.4
+
+- **Columns that normalise to the same name now fail `floe validate`, not the run (#1005).** With
+  `normalize_columns`, a column `order_id` and a JSON or XML selector `order.id` (or `order/id`) both become
+  `order_id`; `floe validate` passed and `floe run` then failed. Validate now rejects it, and the message names both
+  columns or selectors and the shared name.
+
+- **`add_columns` schema evolution on Iceberg is reported and emitted.** An entity run that adds columns to
+  the table now sets `schema_evolution.applied` and lists them in `schema_evolution.added_columns` in `run.json`,
+  emits the `schema_evolution_applied` event, and `schema_evolution.enabled` is `true` whenever the mode is
+  `add_columns`. They were always `false` and the event was never emitted. (#994)
+
+- **Accepted rows now reach the sink chunk by chunk (#1056)**: an entity no longer holds all its accepted rows
+  in memory before the write. The Parquet sink writes parts as the input is read, and the Iceberg sink writes
+  data files into the same single commit. A file's chunks after the first wait on local disk until the file
+  is read to the end, so a late read error or an `abort` still drops all of that file's rows. `append`
+  stages its Parquet parts like `overwrite` does, so a killed run leaves no visible part for a retry to duplicate.
+
+- **`unique` checks with `append` read the target's keys once per entity (#1057)**: an Iceberg table's key columns
+  were scanned once per chunk, and a Parquet sink's parts were loaded into memory in full, so cost and memory
+  grew with chunks times history. The key columns are now written once to a local file in the entity's temp
+  directory, and each chunk joins its keys against that file. Flags and counts are unchanged; the temp
+  directory needs room for the history's key columns.
+
+## v0.8.3
+
+- **`-p env:NAME` (#1068).** `-p` / `--profile` and `profile_path=` accept `env:NAME`, which reads the profile YAML
+  from environment variable `NAME`, so a `kubernetes_job` run needs no profile file in the Job (part of #1047).
+  An unset or empty variable is a `FloeConfigError` naming it. Relative paths a profile supplies resolve against
+  the config's directory, as for a file profile. `execution.runner` stays validate-only.
+- **A Parquet or CSV `overwrite` keeps the previous parts until the new ones are written (#1054)**: the
+  new parts are staged as `_staging-*` files, and the old parts are removed only after every new part
+  was written, so a failed write no longer empties the sink. A crash during the final swap can still
+  leave a mix of old and new parts; see [the Parquet sink](docs/sinks/parquet.md).
+- **Iceberg merges and compaction run DuckDB with `memory_limit=4GB` and a bounded spill directory (#1055).**
+  DuckDB defaulted to 80% of the host's RAM, so a large merge in a Kubernetes pod was OOM-killed
+  instead of spilling. It now caps memory at 4GB and spills to a temporary directory (at most
+  50GB, removed afterwards). Size pods at 6 GiB or more (see `docs/write_modes.md`).
+
 ## Unreleased
 
-### dagster-floe
+### airflow-floe
 
-- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
-  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
+- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
+  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
-- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
-  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
+## dagster-floe v0.4.2
+
+- **Silver lineage events from a `kubernetes_job` run now carry the Dagster parent run (#1076).** The Job
+  container gets `DAGSTER_RUN_ID` and `DAGSTER_JOB_NAME`, and the Job and its pod a `dagster/run-id` label;
+  both `job_scope`s were missing them.
+
+## dagster-floe v0.4.1
+
+**Upgrading from 0.4.0:** the `floe` extra now requires Floe 0.8.3 or newer, and so does the image a
+`kubernetes_job` runs when the run has a profile.
+
+- **A `kubernetes_job` Job gets the profile as content, not as a pod-local path (#1069).** A run with a
+  profile reads it in the run pod and sets it as the Job's `FLOE_PROFILE_CONTENT` env var, with
+  `-p env:FLOE_PROFILE_CONTENT` in place of `-p <path>`; `${VAR}` references stay unexpanded and a remote
+  profile (`s3://...`) is passed as is. The Job image needs Floe 0.8.3 or newer for these runs.
+
+- **A `kubernetes_job` Job is deleted, with its pods, when the run stops watching it (#1070).** An
+  interrupt (a cancelled Dagster run), any error while polling, or the poll timeout used to leave the Job
+  writing tables, so a re-run could collide with it. A Job that reached a final state is left alone, so
+  `ttl_seconds_after_finished` still controls how long its logs stay.
+
+## dagster-floe v0.4.0
+
+**Upgrading from 0.3:** rejected assets move to new keys (below). Works with Floe 0.8.1 or newer; a Glue
+`database` with `{domain}` needs Floe 0.8.2.
 
 - **Breaking: rejected assets are keyed `<domain>/rejected/<table>` (#1046).** They were
   `<domain>/<table>_rejected`, which collided with a real table named `<table>_rejected`. Upgrade:
   the history of the rejected assets restarts under the new key, and any selection, sensor or
   automation condition that names the old keys must be updated.
 
-- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
-  names the database the run writes.
+- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
+  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
 
-### airflow-floe
+- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
+  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
 
-- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
-  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
+- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
+  names the database the run writes.
 
 ## v0.8.2
 

{floe_python-0.8.2 → floe_python-0.8.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.2
+Version: 0.8.4
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

floe_python-0.8.4/conformance/config/val2_normalize_selector_collision/config.yml

@@ -0,0 +1,21 @@
+version: '0.1'
+entities:
+- name: orders
+  source:
+    format: json
+    path: in/orders.json
+  sink:
+    accepted:
+      format: parquet
+      path: out/accepted
+  policy:
+    severity: warn
+  schema:
+    normalize_columns:
+      enabled: true
+    columns:
+    - name: order_id
+      type: string
+    - name: order_ref
+      type: string
+      source: order.id

floe_python-0.8.4/conformance/config/val2_normalize_selector_collision/expected/validate.json

@@ -0,0 +1,4 @@
+{
+  "ok": false,
+  "error": "entity.name=orders column source collision: order_id and order.id -> order_id"
+}

{floe_python-0.8.2 → floe_python-0.8.4}/context/architecture.md

@@ -59,7 +59,7 @@ The CLI and the Python API share the same functions: `_cli.py` parses arguments
 |---|---|
 | Polars | Reading (CSV, JSON, Parquet, Avro), casting, checks, the accepted/rejected split, Parquet and CSV output |
 | PyIceberg (+ `pyiceberg-core`) | Polaris REST catalog, table creation, schema evolution, overwrite/append commits, maintenance |
-| DuckDB (pinned) | Iceberg `MERGE INTO`, unique checks against table history, compaction; spills to disk |
+| DuckDB (pinned) | Iceberg `MERGE INTO`, compaction; spills to disk |
 | pyarrow | Arrow exchange between the three, ORC reads |
 | fastexcel | XLSX |
 | boto3 | S3 storage client |
@@ -80,8 +80,8 @@ them (see `context/decisions/polars-and-arrow.md`).
    → reject_file: copy the raw file to the rejected location, next file
 
 3. Row-level checks (per file, in bounded chunks)
-   cast to declared types → not_null → unique (within the run; with `append`, seeded from
-   the accepted sink) → rows with errors get `__floe_row_index` + `__floe_errors`
+   cast to declared types → not_null → unique (within the run; with `append`, against the
+   accepted sink's key columns, cached once per entity in the entity's temp dir and joined per chunk) → rows with errors get `__floe_row_index` + `__floe_errors`
    → severity decides: warn keeps them, reject routes them to rejected CSV, abort fails the entity
 
 4. Write outputs (once per entity)

{floe_python-0.8.2 → floe_python-0.8.4}/docs/cli.md

@@ -19,7 +19,8 @@
     `image` manifest or the current directory for a `cli` one. An `s3://` `report_base_uri` writes the
     reports there.
   - `-p` applies an environment profile (ignored with `--manifest`: the manifest is already compiled).
-    It takes a local path or an `s3://` URI, read as `--manifest` is (every command that takes `-p` does the same).
+    It takes a local path, an `s3://` URI read as `--manifest` is, or `env:NAME`, the profile YAML held in environment
+    variable `NAME` (every command that takes `-p` does the same).
   - `--domain` selects every entity of the named domains (repeatable or comma-separated), in config order.
     An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state

{floe_python-0.8.2 → floe_python-0.8.4}/docs/config.md

@@ -522,7 +522,8 @@ See `docs/pii.md` for full strategy reference and masking pattern syntax.
   - `enabled`: boolean toggle.
   - `strategy`: `snake_case`, `lower`, `camel_case`, `none`.
   - When enabled, both schema column names and input column names are normalized
-    before checks. If normalization causes a name collision, the run fails.
+    before checks. Two schema columns (or selectors) that normalise to the same name fail `floe validate`; an input column
+    that collides with another after normalization fails the run.
 - `schema_evolution` (optional, v0.2)
   - Omitted by default, Floe behaves as `mode: strict` and `on_incompatible: fail`.
   - `mode`: `strict` or `add_columns`.

{floe_python-0.8.2 → floe_python-0.8.4}/docs/how-it-works.md

@@ -7,9 +7,11 @@ happened in machine-readable reports.
 
 The runtime is a lightweight single-node Python package. Floe reads and validates
 with Polars, writes Iceberg tables through PyIceberg (one catalog commit per
-entity), and runs Iceberg merges and unique checks against the table's history
-through DuckDB, which spills to disk. The libraries exchange data as Arrow, so
-no extra serialization step sits between them.
+entity), and runs Iceberg merges through DuckDB, which spills to disk. With
+`append`, a unique check against the target's history reads the target's key
+columns once per entity into a local file, and each chunk checks its keys
+against that file. The libraries exchange data as Arrow, so no extra
+serialization step sits between them.
 
 This document describes the canonical ingestion pipeline Floe uses for each
 entity. The order is deterministic and is reflected in reports.

{floe_python-0.8.2 → floe_python-0.8.4}/docs/logging.md

@@ -28,13 +28,11 @@ The final line on stdout is always the `run_finished` event.
 - `entity_started`: entity name (`name`, `ts_ms`)
 - `file_started`: entity/input pair (`entity`, `input`, `ts_ms`)
 - `file_finished`: file outcome and counts (`status`, `rows`, `accepted`, `rejected`, `elapsed_ms`, `ts_ms`)
-- `schema_evolution_applied`: additive schema-evolution audit event (`entity`, `mode`, `added_columns`, `ts_ms`); reserved, not emitted in 0.8
+- `schema_evolution_applied`: additive schema-evolution audit event (`entity`, `mode`, `added_columns`, `ts_ms`); emitted once per entity run in which `add_columns` adds columns to an Iceberg table
 - `entity_finished`: entity totals (`status`, `files`, `rows`, `accepted`, `rejected`, `warnings`, `errors`, `ts_ms`)
 - `run_finished`: final run totals and exit code (`status`, `exit_code`, `files`, `rows`, `accepted`, `rejected`, `warnings`, `errors`, `summary_uri`, `ts_ms`)
 - `log`: structured warning/error line (`log_level`, `code`, `message`, optional `entity`, optional `input`, `ts_ms`)
 
-`schema_evolution_applied` is part of the event contract, but 0.8 does not emit it yet, even when `add_columns` adds columns to an Iceberg table.
-
 ### `--log-format off` (default)
 
 - Suppresses all structured event output.

{floe_python-0.8.2 → floe_python-0.8.4}/docs/profiles.md

@@ -51,7 +51,7 @@ All other fields are optional.
 
 ## Profile location
 
-A profile is a local path or an `s3://` URI, for `-p` / `--profile` on every command and for `profile_path=`
+A profile is a local path, an `s3://` URI or `env:NAME`, for `-p` / `--profile` on every command and for `profile_path=`
 in the Python API:
 
 ```bash
@@ -63,6 +63,19 @@ An `s3://` profile is read with the AWS default credential chain (`AWS_ENDPOINT_
 `profile not found: <uri>`. `floe manifest generate -p s3://...` records the URI as `profile_uri`, and the
 SHA-256 of the object as `profile_checksum`. The config (`-c`) is still a local path.
 
+`-p env:NAME` reads the profile YAML from environment variable `NAME`, so a container or Job needs no profile file:
+
+```bash
+FLOE_PROFILE_CONTENT="$(cat profile.yml)" floe run -c config.yml -p env:FLOE_PROFILE_CONTENT
+```
+
+An unset or empty variable fails with a `FloeConfigError` naming it. The profile behaves exactly as a file one
+(`${VAR}` references are expanded the same way). A profile never rebases paths, so a relative path it supplies
+(e.g. through `variables`) resolves against the config's directory, as for a file profile; there is no profile
+directory to resolve against. `execution.runner` is validated only: `floe run` always runs in-process and never launches
+a runner, so a Job started with its own profile does not start another Job. `floe manifest generate -p env:NAME`
+records `env:NAME` as `profile_uri` and the SHA-256 of the variable's content as `profile_checksum`.
+
 ---
 
 ## Variable precedence and resolution

{floe_python-0.8.2 → floe_python-0.8.4}/docs/python-bindings.md

@@ -137,7 +137,7 @@ The callback is **swappable between calls** — calling `set_observer` again rep
 | `run_finished` | `run_id`, `status`, `exit_code`, `files`, `rows`, `accepted`, `rejected`, `summary_uri` |
 | `log` | `run_id`, `log_level`, `code`, `message`, `entity`, `input` |
 
-All events have an `event` key (string discriminator) and `ts_ms` (Unix timestamp in milliseconds). Each event type is exported as a TypedDict (`floe.RunStartedEvent`, …, `floe.LogEvent`; `floe.RunEvent` is their union), defined in [`src/floe/run/events.py`](../src/floe/run/events.py). `schema_evolution_applied` is part of the contract but is not emitted in 0.8.
+All events have an `event` key (string discriminator) and `ts_ms` (Unix timestamp in milliseconds). Each event type is exported as a TypedDict (`floe.RunStartedEvent`, …, `floe.LogEvent`; `floe.RunEvent` is their union), defined in [`src/floe/run/events.py`](../src/floe/run/events.py).
 
 ---
 

{floe_python-0.8.2 → floe_python-0.8.4}/docs/report.md

@@ -106,11 +106,13 @@ Merge-specific metrics (Iceberg `merge_scd1`):
 
 ### `schema_evolution` (entity report)
 
-- `enabled`, `applied`, `incompatible_changes_detected`: always `false` in 0.8
+- `enabled`: `true` when `mode` is `add_columns`
 - `mode`: resolved config mode (`strict` or `add_columns`)
+- `applied`: `true` when Floe added columns to the Iceberg table during the write (D13)
+- `added_columns`: ordered list of the columns added in that write; present only when `applied` is `true`
+- `incompatible_changes_detected`: always `false` in 0.8 (a non-additive change fails the entity before the report)
 
-`mode: add_columns` does add new columns to an Iceberg table (D13), but 0.8 does not record
-them here and emits no `schema_evolution_applied` event yet.
+Floe also emits a `schema_evolution_applied` event when it adds columns (see [Logging](logging.md)).
 
 Notes:
 - Metrics are populated when the writer can collect them cheaply and reliably.

{floe_python-0.8.2 → floe_python-0.8.4}/docs/sinks/parquet.md

@@ -18,7 +18,7 @@ entities:
 Semantics:
 - `sink.accepted.format: parquet` writes one or more part files under `sink.accepted.path`.
 - Write mode comes from `sink.write_mode`:
-  - `overwrite`: remove the existing `part-*.parquet` files, then write `part-00000.parquet`, `part-00001.parquet`, ...
+  - `overwrite`: write the new parts as `part-00000.parquet`, `part-00001.parquet`, ... and remove the old `part-*.parquet` files once every new part was written
   - `append`: add new `part-<uuid>.parquet` files alongside existing ones.
   - `merge_scd1` / `merge_scd2`: not supported for Parquet; use the [Iceberg](iceberg.md) sink.
 
@@ -26,7 +26,10 @@ Semantics:
 
 Large accepted datasets are split into multiple part files. The split threshold is controlled by
 `sink.accepted.options.max_size_per_file` (default 256 MB), measured on the in-memory size of
-the rows. The entity's accepted rows are written once, after every input file is read.
+the rows. Accepted rows are written as the input is read, so memory holds about one part plus one
+chunk of input, not the whole entity. Every part is staged as a `_staging-<id>-part-*` file and
+copied to its `part-*` name only once the whole input was read and written, so a write that fails
+(or a killed run) leaves no visible part of that run; orphaned `_staging-*` files are safe to delete.
 
 ```yaml
 sink:
@@ -60,10 +63,14 @@ with valid output files in the accepted directory.
 
 **Overwrite is not atomic**
 
-`overwrite` deletes the old part files before it writes the new ones. A failure in between
-leaves the directory empty or partly written until the next run. The run report and
-incremental state are not committed in that case, so Floe re-processes the same inputs on the
-next run.
+`overwrite` first writes every new part under a `_staging-<id>-part-NNNNN.parquet` name, which
+readers of `part-*.parquet` do not see. Only when all of them were written does it copy them to
+`part-NNNNN.parquet`, delete the old parts that were not replaced, then delete the staged files.
+A write failure leaves the previous parts untouched (the staged files are removed). A crash or a
+storage error during the final copy and delete steps can leave a mix of new and old parts (rows
+duplicated or missing) and orphaned `_staging-*` files, which are safe to delete. The run report
+and incremental state are not committed in that case, so Floe re-processes the same inputs on
+the next run and a re-run restores a clean directory.
 
 **If you need transactional accepted writes, use the [Iceberg](iceberg.md) sink instead.**
 Each entity's Iceberg write is one atomic catalog commit.

{floe_python-0.8.2 → floe_python-0.8.4}/docs/write_modes.md

@@ -54,6 +54,13 @@ row closes its current version (`valid_to` set, `is_current` false) and inserts
 unchanged rows are left alone. `compare_columns` and `ignore_columns` choose which columns count as a
 change. The report adds `closed_count` and `unchanged_count` to the merge metrics.
 
+### Sizing merge pods
+
+Merges and compaction run DuckDB with `memory_limit=4GB`, spilling to a temporary directory capped at
+`max_temp_directory_size=50GB` (removed when the run ends). DuckDB's resident memory overshoots its limit
+by 0.2-1.9 GiB, more as delete files accumulate, so size the pod at `memory_limit` + 2 GiB (6 GiB), and give
+it ephemeral disk for the spill.
+
 ## Rejected output behavior
 
 `sink.write_mode` is shared with rejected output:

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/README.md

@@ -30,22 +30,30 @@ Current model:
 ## Install
 
 Prereqs:
-- Floe 0.8.1 or newer (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
+- Floe 0.8.3 or newer (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
 - Python 3.10+.
 
 ```bash
 pip install dagster-floe
-pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.1,<0.9 for the in-process runner
+pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.3,<0.9 for the in-process runner
 ```
 
 ## Supported Floe versions
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.2 | 0.8.3 or newer | `0.8` |
+| 0.4.1 | 0.8.3 or newer (`kubernetes_job` runs pass the profile as `-p env:FLOE_PROFILE_CONTENT`) | `0.8` |
+| 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |
 | 0.3.1 | 0.8.1 or newer (`job_scope: domain` Jobs run `floe run --domain`) | `0.8` |
 | 0.3.0 | 0.8 | `0.8` |
 | 0.2.x | 0.7 and earlier | see 0.2.x docs |
 
+With the `kubernetes_job` runner, a run that has a profile (config mode, `profile_path=`) reads the profile in the
+run pod and hands its text to the Job as the `FLOE_PROFILE_CONTENT` env var, with `-p env:FLOE_PROFILE_CONTENT`
+in place of the pod-local path. `${VAR}` references stay unexpanded: the Job expands them. The Job image needs
+Floe 0.8.3 or newer for these runs. A profile that is already a URI such as `s3://...` is passed as is.
+
 ## Development install (from this repo)
 
 ```bash
@@ -158,7 +166,7 @@ From 0.3.0, remove that patch and set the variables above.
 
 ## Lineage integration
 
-The connector automatically injects two environment variables into every floe subprocess:
+The connector automatically injects two environment variables into every floe run it starts locally or as a `kubernetes_job` (not `databricks_job`):
 
 - `DAGSTER_RUN_ID` — the current Dagster run ID
 - `DAGSTER_JOB_NAME` — the Dagster job name

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.3.1"
+version = "0.4.2"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"
@@ -33,7 +33,7 @@ kubernetes = [
   "kubernetes>=28",
 ]
 floe = [
-  "floe-python>=0.8.1,<0.9",
+  "floe-python>=0.8.3,<0.9",
 ]
 remote = [
   "fsspec",

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -561,6 +561,7 @@ def _make_domain_multi_asset(
         run_id = getattr(run, "run_id", None) if run is not None else None
         # concurrent groups of one Dagster run must not share a Floe run id (it names the summary file)
         floe_run_id = f"{run_id}_{context.op_def.name}" if run_id else None
+        dagster_job_name = getattr(run, "job_name", None) if run is not None else None
         names = [entity.name for entity in chosen]
         # the whole domain of the manifest: `floe run --domain` says it; a custom template stays as written
         whole_domain = bool(names) and set(names) == domain_names and execution.per_entity_args == [
@@ -575,6 +576,8 @@ def _make_domain_multi_asset(
             entities=names,
             execution=replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution,
             runner_definition=runner_definition,
+            dagster_run_id=run_id,
+            dagster_job_name=dagster_job_name,
         )
         finished = _parse_finished(context, result, execution, run_id)
         summary = _load_run_summary(context, finished, config_uri)

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/kubernetes_runner.py

@@ -1,8 +1,11 @@
 from __future__ import annotations
 
+import logging
+import re
 import time
 import uuid
 from dataclasses import replace
+from pathlib import Path
 from typing import Any
 
 from .k8s_status import (
@@ -18,8 +21,13 @@ from .k8s_status import (
 from .manifest import ManifestRunnerDefinition
 from .runner import RunResult
 
+_log = logging.getLogger(__name__)
+
+_PROFILE_ENV = "FLOE_PROFILE_CONTENT"
 _DEFAULT_POLL_INTERVAL_S = 10
 _DEFAULT_TIMEOUT_S = 3600
+_RUN_ID_LABEL = "dagster/run-id"
+_LABEL_VALUE = re.compile(r"[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?")
 
 
 def run_kubernetes_job(
@@ -29,13 +37,17 @@ def run_kubernetes_job(
     runner: ManifestRunnerDefinition,
     jobs_api: Any = None,
     core_api: Any = None,
+    dagster_run_id: str | None = None,
+    dagster_job_name: str | None = None,
 ) -> RunResult:
     namespace = runner.namespace
     if not namespace:
         raise ValueError("kubernetes_job runner requires a non-empty 'namespace' field")
 
     job_name = _make_job_name(entity)
-    job_spec = build_k8s_job_spec(job_name, cmd_args, runner)
+    job_spec = build_k8s_job_spec(
+        job_name, cmd_args, runner, dagster_run_id=dagster_run_id, dagster_job_name=dagster_job_name
+    )
 
     poll_interval = runner.poll_interval_seconds or _DEFAULT_POLL_INTERVAL_S
     timeout = runner.timeout_seconds or _DEFAULT_TIMEOUT_S
@@ -50,19 +62,26 @@ def run_kubernetes_job(
     deadline = time.monotonic() + timeout
     status = STATUS_SUBMITTED
     last_job: Any = None
+    reached_final_state = False
 
-    while True:
-        last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
-        status = map_k8s_job_status(last_job)
-        if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
-            break
-        if time.monotonic() >= deadline:
-            status = STATUS_TIMEOUT
-            break
-        time.sleep(poll_interval)
-
-    stdout = _collect_job_logs(core_api, job_name, namespace)
-    pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    try:
+        while True:
+            last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
+            status = map_k8s_job_status(last_job)
+            if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
+                reached_final_state = True
+                break
+            if time.monotonic() >= deadline:
+                status = STATUS_TIMEOUT
+                break
+            time.sleep(poll_interval)
+        # before any deletion: with background propagation the pod would be gone
+        stdout = _collect_job_logs(core_api, job_name, namespace)
+        pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    finally:
+        # an interrupt, an error or the poll timeout leaves a Job nobody watches: it would keep writing
+        if not reached_final_state:
+            _delete_job(jobs_api, job_name, namespace)
 
     failure_reason = _resolve_failure_reason(status=status, job=last_job, pod_ctx=pod_ctx)
     if status == STATUS_SUCCEEDED:
@@ -92,7 +111,22 @@ def run_kubernetes_job(
     )
 
 
-def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunnerDefinition) -> dict[str, Any]:
+def _delete_job(jobs_api: Any, job_name: str, namespace: str) -> None:
+    """Best effort, with its pods: a failure is logged and must not mask the error being raised."""
+    try:
+        jobs_api.delete_namespaced_job(job_name, namespace, propagation_policy="Background")
+    except Exception:
+        _log.warning("could not delete Kubernetes Job %s/%s", namespace, job_name, exc_info=True)
+
+
+def build_k8s_job_spec(
+    job_name: str,
+    cmd_args: list[str],
+    runner: ManifestRunnerDefinition,
+    *,
+    dagster_run_id: str | None = None,
+    dagster_job_name: str | None = None,
+) -> dict[str, Any]:
     if not runner.image:
         raise ValueError("kubernetes_job runner requires a non-empty 'image' field")
     if not runner.namespace:
@@ -102,6 +136,7 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     container_args: list[str] = list(cmd_args[1:]) if len(cmd_args) > 1 else []
     if runner.command is not None:
         container_cmd = runner.command
+    container_args, profile_text = _inline_profile(container_args)
     if runner.args is not None:  # a prefix: replacing the list would drop the entity selection
         container_args = runner.args + container_args
 
@@ -123,6 +158,15 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
                 }
             )
 
+    # the lineage parent run (`floe.lineage.events.parent`): the Job has no Dagster process env
+    if dagster_run_id:
+        env_vars.append({"name": "DAGSTER_RUN_ID", "value": dagster_run_id})
+    if dagster_job_name:
+        env_vars.append({"name": "DAGSTER_JOB_NAME", "value": dagster_job_name})
+
+    if profile_text is not None:
+        env_vars.append({"name": _PROFILE_ENV, "value": profile_text})
+
     container: dict[str, Any] = {"name": "floe-runner", "image": runner.image}
     if container_cmd:
         container["command"] = container_cmd
@@ -152,8 +196,10 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
         pod_spec["tolerations"] = runner.tolerations
 
     # the defaults win: `floe-job` is how the runner finds the pod again
+    run_label = {_RUN_ID_LABEL: dagster_run_id} if dagster_run_id and _LABEL_VALUE.fullmatch(dagster_run_id) else {}
+    job_labels = {"app": "floe-runner", **run_label}
     pod_metadata: dict[str, Any] = {
-        "labels": {**(runner.labels or {}), "app": "floe-runner", "floe-job": job_name}
+        "labels": {**(runner.labels or {}), **job_labels, "floe-job": job_name}
     }
     if runner.annotations:
         pod_metadata["annotations"] = runner.annotations
@@ -173,11 +219,22 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     return {
         "apiVersion": "batch/v1",
         "kind": "Job",
-        "metadata": {"name": job_name, "namespace": runner.namespace, "labels": {"app": "floe-runner"}},
+        "metadata": {"name": job_name, "namespace": runner.namespace, "labels": job_labels},
         "spec": job_spec,
     }
 
 
+def _inline_profile(args: list[str]) -> tuple[list[str], str | None]:
+    """A profile path only exists on the run pod's disk: the Job gets its text (`${VAR}` unexpanded) instead."""
+    for i, arg in enumerate(args[:-1]):
+        if arg in ("-p", "--profile"):
+            path = args[i + 1]
+            if "://" in path or path.startswith("env:"):  # already reachable from the Job
+                return args, None
+            return [*args[:i], "-p", f"env:{_PROFILE_ENV}", *args[i + 2 :]], Path(path).read_text(encoding="utf-8")
+    return args, None
+
+
 def _resource_requests(resources: dict[str, Any] | None) -> dict[str, str]:
     requests: dict[str, str] = {}
     if resources:

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/src/floe_dagster/runner.py

@@ -32,6 +32,8 @@ class Runner:
         execution: ManifestExecution,
         runner_definition: ManifestRunnerDefinition,
         manifest_uri: str | None = None,
+        dagster_run_id: str | None = None,
+        dagster_job_name: str | None = None,
     ) -> RunResult:
         """One run for several entities of `domain`, one after another (`job_scope: domain`)."""
         raise NotImplementedError
@@ -64,6 +66,8 @@ class LocalRunner(Runner):
         execution: ManifestExecution,
         runner_definition: ManifestRunnerDefinition,
         manifest_uri: str | None = None,
+        dagster_run_id: str | None = None,
+        dagster_job_name: str | None = None,
     ) -> RunResult:
         if runner_definition.runner_type != "kubernetes_job":
             raise NotImplementedError("job_scope: domain needs a kubernetes_job runner")
@@ -81,7 +85,11 @@ class LocalRunner(Runner):
         from .kubernetes_runner import run_kubernetes_job, sized_for_domain
 
         return run_kubernetes_job(
-            args, entity=domain, runner=sized_for_domain(runner_definition, domain)
+            args,
+            entity=domain,
+            runner=sized_for_domain(runner_definition, domain),
+            dagster_run_id=dagster_run_id,
+            dagster_job_name=dagster_job_name,
         )
 
     def run_floe_entity(
@@ -110,7 +118,13 @@ class LocalRunner(Runner):
             )
             from .kubernetes_runner import run_kubernetes_job
 
-            return run_kubernetes_job(args, entity=entity, runner=runner_definition)
+            return run_kubernetes_job(
+                args,
+                entity=entity,
+                runner=runner_definition,
+                dagster_run_id=run_id,
+                dagster_job_name=dagster_job_name,
+            )
 
         if runner_definition is not None and runner_definition.runner_type == "databricks_job":
             if execution is None:

{floe_python-0.8.2 → floe_python-0.8.4}/orchestrators/dagster-floe/tests/test_domain_scope.py

@@ -92,6 +92,16 @@ def test_domain_scope_runs_the_entities_in_one_job_with_a_materialization_each(t
     assert args[args.index("--run-id") + 1].startswith(f"{result.run_id}_floe_")  # one Floe run id per group
 
 
+def test_domain_scope_job_carries_the_dagster_run_for_lineage(tmp_path: Path) -> None:
+    result, jobs_api = _run(tmp_path, dict.fromkeys(NAMES, "success"))
+
+    body = jobs_api.create_namespaced_job.call_args.kwargs["body"]
+    env = {e["name"]: e.get("value") for e in body["spec"]["template"]["spec"]["containers"][0]["env"]}
+    assert env["DAGSTER_RUN_ID"] == result.run_id
+    assert env["DAGSTER_JOB_NAME"]
+    assert body["metadata"]["labels"]["dagster/run-id"] == result.run_id
+
+
 def test_domain_scope_subset_runs_only_the_selected_entities(tmp_path: Path) -> None:
     selection = [AssetKey(["sales", "orders"]), AssetKey(["sales", "refunds"])]
     result, jobs_api = _run(tmp_path, {"orders": "success", "refunds": "success"}, selection=selection)