floe-python 0.8.2__tar.gz → 0.8.3__tar.gz

{floe_python-0.8.2 → floe_python-0.8.3}/CHANGELOG.md

@@ -2,28 +2,58 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.3
+
+- **`-p env:NAME` (#1068).** `-p` / `--profile` and `profile_path=` accept `env:NAME`, which reads the profile YAML
+  from environment variable `NAME`, so a `kubernetes_job` run needs no profile file in the Job (part of #1047).
+  An unset or empty variable is a `FloeConfigError` naming it. Relative paths a profile supplies resolve against
+  the config's directory, as for a file profile. `execution.runner` stays validate-only.
+- **A Parquet or CSV `overwrite` keeps the previous parts until the new ones are written (#1054)**: the
+  new parts are staged as `_staging-*` files, and the old parts are removed only after every new part
+  was written, so a failed write no longer empties the sink. A crash during the final swap can still
+  leave a mix of old and new parts; see [the Parquet sink](docs/sinks/parquet.md).
+- **Iceberg merges and compaction run DuckDB with `memory_limit=4GB` and a bounded spill directory (#1055).**
+  DuckDB defaulted to 80% of the host's RAM, so a large merge in a Kubernetes pod was OOM-killed
+  instead of spilling. It now caps memory at 4GB and spills to a temporary directory (at most
+  50GB, removed afterwards). Size pods at 6 GiB or more (see `docs/write_modes.md`).
+
 ## Unreleased
 
 ### dagster-floe
 
-- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
-  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
+- **A `kubernetes_job` Job gets the profile as content, not as a pod-local path (#1069).** A run with a
+  profile reads it in the run pod and sets it as the Job's `FLOE_PROFILE_CONTENT` env var, with
+  `-p env:FLOE_PROFILE_CONTENT` in place of `-p <path>`; `${VAR}` references stay unexpanded and a remote
+  profile (`s3://...`) is passed as is. The Job image needs Floe 0.8.3 or newer for these runs.
 
-- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
-  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
+- **A `kubernetes_job` Job is deleted, with its pods, when the run stops watching it (#1070).** An
+  interrupt (a cancelled Dagster run), any error while polling, or the poll timeout used to leave the Job
+  writing tables, so a re-run could collide with it. A Job that reached a final state is left alone, so
+  `ttl_seconds_after_finished` still controls how long its logs stay.
+
+### airflow-floe
+
+- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
+  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
+
+## dagster-floe v0.4.0
+
+**Upgrading from 0.3:** rejected assets move to new keys (below). Works with Floe 0.8.1 or newer; a Glue
+`database` with `{domain}` needs Floe 0.8.2.
 
 - **Breaking: rejected assets are keyed `<domain>/rejected/<table>` (#1046).** They were
   `<domain>/<table>_rejected`, which collided with a real table named `<table>_rejected`. Upgrade:
   the history of the rejected assets restarts under the new key, and any selection, sensor or
   automation condition that names the old keys must be updated.
 
-- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
-  names the database the run writes.
+- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
+  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
 
-### airflow-floe
+- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
+  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
 
-- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
-  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
+- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
+  names the database the run writes.
 
 ## v0.8.2
 

{floe_python-0.8.2 → floe_python-0.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.2
+Version: 0.8.3
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.2 → floe_python-0.8.3}/docs/cli.md

@@ -19,7 +19,8 @@
     `image` manifest or the current directory for a `cli` one. An `s3://` `report_base_uri` writes the
     reports there.
   - `-p` applies an environment profile (ignored with `--manifest`: the manifest is already compiled).
-    It takes a local path or an `s3://` URI, read as `--manifest` is (every command that takes `-p` does the same).
+    It takes a local path, an `s3://` URI read as `--manifest` is, or `env:NAME`, the profile YAML held in environment
+    variable `NAME` (every command that takes `-p` does the same).
   - `--domain` selects every entity of the named domains (repeatable or comma-separated), in config order.
     An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state

{floe_python-0.8.2 → floe_python-0.8.3}/docs/profiles.md

@@ -51,7 +51,7 @@ All other fields are optional.
 
 ## Profile location
 
-A profile is a local path or an `s3://` URI, for `-p` / `--profile` on every command and for `profile_path=`
+A profile is a local path, an `s3://` URI or `env:NAME`, for `-p` / `--profile` on every command and for `profile_path=`
 in the Python API:
 
 ```bash
@@ -63,6 +63,19 @@ An `s3://` profile is read with the AWS default credential chain (`AWS_ENDPOINT_
 `profile not found: <uri>`. `floe manifest generate -p s3://...` records the URI as `profile_uri`, and the
 SHA-256 of the object as `profile_checksum`. The config (`-c`) is still a local path.
 
+`-p env:NAME` reads the profile YAML from environment variable `NAME`, so a container or Job needs no profile file:
+
+```bash
+FLOE_PROFILE_CONTENT="$(cat profile.yml)" floe run -c config.yml -p env:FLOE_PROFILE_CONTENT
+```
+
+An unset or empty variable fails with a `FloeConfigError` naming it. The profile behaves exactly as a file one
+(`${VAR}` references are expanded the same way). A profile never rebases paths, so a relative path it supplies
+(e.g. through `variables`) resolves against the config's directory, as for a file profile; there is no profile
+directory to resolve against. `execution.runner` is validated only: `floe run` always runs in-process and never launches
+a runner, so a Job started with its own profile does not start another Job. `floe manifest generate -p env:NAME`
+records `env:NAME` as `profile_uri` and the SHA-256 of the variable's content as `profile_checksum`.
+
 ---
 
 ## Variable precedence and resolution

{floe_python-0.8.2 → floe_python-0.8.3}/docs/sinks/parquet.md

@@ -18,7 +18,7 @@ entities:
 Semantics:
 - `sink.accepted.format: parquet` writes one or more part files under `sink.accepted.path`.
 - Write mode comes from `sink.write_mode`:
-  - `overwrite`: remove the existing `part-*.parquet` files, then write `part-00000.parquet`, `part-00001.parquet`, ...
+  - `overwrite`: write the new parts as `part-00000.parquet`, `part-00001.parquet`, ... and remove the old `part-*.parquet` files once every new part was written
   - `append`: add new `part-<uuid>.parquet` files alongside existing ones.
   - `merge_scd1` / `merge_scd2`: not supported for Parquet; use the [Iceberg](iceberg.md) sink.
 
@@ -60,10 +60,14 @@ with valid output files in the accepted directory.
 
 **Overwrite is not atomic**
 
-`overwrite` deletes the old part files before it writes the new ones. A failure in between
-leaves the directory empty or partly written until the next run. The run report and
-incremental state are not committed in that case, so Floe re-processes the same inputs on the
-next run.
+`overwrite` first writes every new part under a `_staging-<id>-part-NNNNN.parquet` name, which
+readers of `part-*.parquet` do not see. Only when all of them were written does it copy them to
+`part-NNNNN.parquet`, delete the old parts that were not replaced, then delete the staged files.
+A write failure leaves the previous parts untouched (the staged files are removed). A crash or a
+storage error during the final copy and delete steps can leave a mix of new and old parts (rows
+duplicated or missing) and orphaned `_staging-*` files, which are safe to delete. The run report
+and incremental state are not committed in that case, so Floe re-processes the same inputs on
+the next run and a re-run restores a clean directory.
 
 **If you need transactional accepted writes, use the [Iceberg](iceberg.md) sink instead.**
 Each entity's Iceberg write is one atomic catalog commit.

{floe_python-0.8.2 → floe_python-0.8.3}/docs/write_modes.md

@@ -54,6 +54,13 @@ row closes its current version (`valid_to` set, `is_current` false) and inserts
 unchanged rows are left alone. `compare_columns` and `ignore_columns` choose which columns count as a
 change. The report adds `closed_count` and `unchanged_count` to the merge metrics.
 
+### Sizing merge pods
+
+Merges and compaction run DuckDB with `memory_limit=4GB`, spilling to a temporary directory capped at
+`max_temp_directory_size=50GB` (removed when the run ends). DuckDB's resident memory overshoots its limit
+by 0.2-1.9 GiB, more as delete files accumulate, so size the pod at `memory_limit` + 2 GiB (6 GiB), and give
+it ephemeral disk for the spill.
+
 ## Rejected output behavior
 
 `sink.write_mode` is shared with rejected output:

{floe_python-0.8.2 → floe_python-0.8.3}/orchestrators/dagster-floe/README.md

@@ -42,10 +42,16 @@ pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.1,<0.9 for t
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |
 | 0.3.1 | 0.8.1 or newer (`job_scope: domain` Jobs run `floe run --domain`) | `0.8` |
 | 0.3.0 | 0.8 | `0.8` |
 | 0.2.x | 0.7 and earlier | see 0.2.x docs |
 
+With the `kubernetes_job` runner, a run that has a profile (config mode, `profile_path=`) reads the profile in the
+run pod and hands its text to the Job as the `FLOE_PROFILE_CONTENT` env var, with `-p env:FLOE_PROFILE_CONTENT`
+in place of the pod-local path. `${VAR}` references stay unexpanded: the Job expands them. The Job image needs
+Floe 0.8.3 or newer for these runs. A profile that is already a URI such as `s3://...` is passed as is.
+
 ## Development install (from this repo)
 
 ```bash

{floe_python-0.8.2 → floe_python-0.8.3}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.3.1"
+version = "0.4.0"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"

{floe_python-0.8.2 → floe_python-0.8.3}/orchestrators/dagster-floe/src/floe_dagster/kubernetes_runner.py

@@ -1,8 +1,10 @@
 from __future__ import annotations
 
+import logging
 import time
 import uuid
 from dataclasses import replace
+from pathlib import Path
 from typing import Any
 
 from .k8s_status import (
@@ -18,6 +20,9 @@ from .k8s_status import (
 from .manifest import ManifestRunnerDefinition
 from .runner import RunResult
 
+_log = logging.getLogger(__name__)
+
+_PROFILE_ENV = "FLOE_PROFILE_CONTENT"
 _DEFAULT_POLL_INTERVAL_S = 10
 _DEFAULT_TIMEOUT_S = 3600
 
@@ -50,19 +55,26 @@ def run_kubernetes_job(
     deadline = time.monotonic() + timeout
     status = STATUS_SUBMITTED
     last_job: Any = None
+    reached_final_state = False
 
-    while True:
-        last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
-        status = map_k8s_job_status(last_job)
-        if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
-            break
-        if time.monotonic() >= deadline:
-            status = STATUS_TIMEOUT
-            break
-        time.sleep(poll_interval)
-
-    stdout = _collect_job_logs(core_api, job_name, namespace)
-    pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    try:
+        while True:
+            last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
+            status = map_k8s_job_status(last_job)
+            if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
+                reached_final_state = True
+                break
+            if time.monotonic() >= deadline:
+                status = STATUS_TIMEOUT
+                break
+            time.sleep(poll_interval)
+        # before any deletion: with background propagation the pod would be gone
+        stdout = _collect_job_logs(core_api, job_name, namespace)
+        pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    finally:
+        # an interrupt, an error or the poll timeout leaves a Job nobody watches: it would keep writing
+        if not reached_final_state:
+            _delete_job(jobs_api, job_name, namespace)
 
     failure_reason = _resolve_failure_reason(status=status, job=last_job, pod_ctx=pod_ctx)
     if status == STATUS_SUCCEEDED:
@@ -92,6 +104,14 @@ def run_kubernetes_job(
     )
 
 
+def _delete_job(jobs_api: Any, job_name: str, namespace: str) -> None:
+    """Best effort, with its pods: a failure is logged and must not mask the error being raised."""
+    try:
+        jobs_api.delete_namespaced_job(job_name, namespace, propagation_policy="Background")
+    except Exception:
+        _log.warning("could not delete Kubernetes Job %s/%s", namespace, job_name, exc_info=True)
+
+
 def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunnerDefinition) -> dict[str, Any]:
     if not runner.image:
         raise ValueError("kubernetes_job runner requires a non-empty 'image' field")
@@ -102,6 +122,7 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     container_args: list[str] = list(cmd_args[1:]) if len(cmd_args) > 1 else []
     if runner.command is not None:
         container_cmd = runner.command
+    container_args, profile_text = _inline_profile(container_args)
     if runner.args is not None:  # a prefix: replacing the list would drop the entity selection
         container_args = runner.args + container_args
 
@@ -123,6 +144,9 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
                 }
             )
 
+    if profile_text is not None:
+        env_vars.append({"name": _PROFILE_ENV, "value": profile_text})
+
     container: dict[str, Any] = {"name": "floe-runner", "image": runner.image}
     if container_cmd:
         container["command"] = container_cmd
@@ -178,6 +202,17 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     }
 
 
+def _inline_profile(args: list[str]) -> tuple[list[str], str | None]:
+    """A profile path only exists on the run pod's disk: the Job gets its text (`${VAR}` unexpanded) instead."""
+    for i, arg in enumerate(args[:-1]):
+        if arg in ("-p", "--profile"):
+            path = args[i + 1]
+            if "://" in path or path.startswith("env:"):  # already reachable from the Job
+                return args, None
+            return [*args[:i], "-p", f"env:{_PROFILE_ENV}", *args[i + 2 :]], Path(path).read_text(encoding="utf-8")
+    return args, None
+
+
 def _resource_requests(resources: dict[str, Any] | None) -> dict[str, str]:
     requests: dict[str, str] = {}
     if resources:

{floe_python-0.8.2 → floe_python-0.8.3}/orchestrators/dagster-floe/tests/test_kubernetes_runner.py

@@ -5,6 +5,9 @@ from pathlib import Path
 from types import SimpleNamespace
 from unittest.mock import MagicMock, patch
 
+import pytest
+from dagster._core.errors import DagsterExecutionInterruptedError
+
 from floe_dagster.kubernetes_runner import (
     STATUS_FAILED,
     STATUS_RUNNING,
@@ -409,3 +412,96 @@ def test_a_failed_job_reports_the_latest_attempt() -> None:
         result = _run_with_pods(ordered, job)
         assert result.stdout == "logs of attempt-3"
         assert result.backend_metadata["pod_name"] == "attempt-3"
+
+
+@pytest.mark.parametrize("interrupt", [DagsterExecutionInterruptedError, KeyboardInterrupt])
+def test_an_interrupted_poll_deletes_the_job(interrupt) -> None:
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = _job(active=1)
+    with patch("time.sleep", side_effect=interrupt), pytest.raises(interrupt):
+        run_kubernetes_job(["floe", "run"], entity="orders", runner=_runner(), jobs_api=jobs_api, core_api=core_api)
+    name = jobs_api.create_namespaced_job.call_args.kwargs["body"]["metadata"]["name"]
+    jobs_api.delete_namespaced_job.assert_called_once_with(name, "data", propagation_policy="Background")
+
+
+def test_a_failing_deletion_does_not_mask_the_interrupt() -> None:
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = _job(active=1)
+    jobs_api.delete_namespaced_job.side_effect = RuntimeError("forbidden")
+    with patch("time.sleep", side_effect=KeyboardInterrupt), pytest.raises(KeyboardInterrupt):
+        run_kubernetes_job(["floe", "run"], entity="orders", runner=_runner(), jobs_api=jobs_api, core_api=core_api)
+    jobs_api.delete_namespaced_job.assert_called_once()
+
+
+def test_a_poll_timeout_collects_logs_then_deletes_the_job_and_still_reports_timeout() -> None:
+    order: list[str] = []
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = _job(active=1)
+    jobs_api.delete_namespaced_job.side_effect = lambda *a, **k: order.append("delete")
+    pod = MagicMock()
+    pod.metadata.name = "pod-1"
+    core_api.list_namespaced_pod.return_value = MagicMock(items=[pod])
+    core_api.read_namespaced_pod_log.side_effect = lambda **k: order.append("logs") or "last words"
+    with patch("time.sleep"), patch("time.monotonic", side_effect=[0, 61]):
+        result = run_kubernetes_job(["floe", "run"], entity="orders", runner=_runner(), jobs_api=jobs_api, core_api=core_api)
+    assert result.status == STATUS_TIMEOUT
+    assert result.stdout == "last words"
+    assert result.backend_metadata["pod_name"] == "pod-1"
+    assert order == ["logs", "delete"]
+    name = jobs_api.create_namespaced_job.call_args.kwargs["body"]["metadata"]["name"]
+    jobs_api.delete_namespaced_job.assert_called_once_with(name, "data", propagation_policy="Background")
+
+
+@pytest.mark.parametrize(
+    "job",
+    [
+        _job(succeeded=1, conditions=[{"type": "Complete", "status": "True"}]),
+        _job(failed=1, conditions=[{"type": "Failed", "status": "True", "reason": "BackoffLimitExceeded"}]),
+        _job(failed=1, conditions=[{"type": "Failed", "status": "True", "reason": "DeadlineExceeded"}]),
+    ],
+)
+def test_a_job_in_a_final_state_is_not_deleted(job) -> None:
+    jobs_api, core_api = MagicMock(), MagicMock()
+    jobs_api.read_namespaced_job.return_value = job
+    core_api.list_namespaced_pod.return_value = MagicMock(items=[])
+    run_kubernetes_job(["floe", "run"], entity="orders", runner=_runner(), jobs_api=jobs_api, core_api=core_api)
+    jobs_api.delete_namespaced_job.assert_not_called()
+
+
+def _profile_container(tmp_path: Path, args_tail: list[str], profile_text: str = "") -> dict:
+    profile = tmp_path / "profile.yml"
+    profile.write_text(profile_text, encoding="utf-8")
+    args = ["floe", "run", "-c", "cfg.yml", *[str(profile) if a == "<profile>" else a for a in args_tail]]
+    spec = build_k8s_job_spec("floe-orders-abc", args, _runner())
+    return spec["spec"]["template"]["spec"]["containers"][0]
+
+
+def test_a_profile_path_is_passed_to_the_job_as_content(tmp_path: Path) -> None:
+    text = "version: 1\nstorage:\n  secret: ${SECRET}\n"
+    container = _profile_container(tmp_path, ["-p", "<profile>", "--entities", "orders"], text)
+    assert container["args"] == ["run", "-c", "cfg.yml", "-p", "env:FLOE_PROFILE_CONTENT", "--entities", "orders"]
+    assert str(tmp_path) not in " ".join(container["args"])
+    env = {e["name"]: e for e in container["env"]}
+    assert env["FLOE_PROFILE_CONTENT"] == {"name": "FLOE_PROFILE_CONTENT", "value": text}  # ${SECRET} unexpanded
+    assert env["LOG_LEVEL"]["value"] == "INFO"
+
+
+def test_the_long_profile_flag_is_replaced_too(tmp_path: Path) -> None:
+    container = _profile_container(tmp_path, ["--profile", "<profile>"], "a: 1\n")
+    assert container["args"] == ["run", "-c", "cfg.yml", "-p", "env:FLOE_PROFILE_CONTENT"]
+    assert {"name": "FLOE_PROFILE_CONTENT", "value": "a: 1\n"} in container["env"]
+
+
+def test_a_remote_profile_passes_through(tmp_path: Path) -> None:
+    args = ["floe", "run", "-c", "cfg.yml", "-p", "s3://bucket/profile.yml"]
+    container = build_k8s_job_spec("floe-orders-abc", args, _runner())["spec"]["template"]["spec"]["containers"][0]
+    assert container["args"] == ["run", "-c", "cfg.yml", "-p", "s3://bucket/profile.yml"]
+    assert "FLOE_PROFILE_CONTENT" not in {e["name"] for e in container["env"]}
+
+
+def test_a_run_without_a_profile_is_unchanged() -> None:
+    container = build_k8s_job_spec("floe-orders-abc", ["floe", "run", "-c", "cfg.yml"], _runner())["spec"]["template"][
+        "spec"
+    ]["containers"][0]
+    assert container["args"] == ["run", "-c", "cfg.yml"]
+    assert container["env"] == [{"name": "LOG_LEVEL", "value": "INFO"}]

{floe_python-0.8.2 → floe_python-0.8.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.2"
+version = "0.8.3"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/_cli.py

@@ -40,7 +40,7 @@ def _realpath(path: str) -> Path:
 def _profile(location: str | None) -> ProfileConfig | None:
     if not location:
         return None
-    return parse_profile(location if location.startswith("s3://") else _realpath(location))  # fmt: skip
+    return parse_profile(location if location.startswith(("s3://", "env:")) else _realpath(location))  # fmt: skip
 
 
 def _load(path: Path, profile: ProfileConfig | None) -> RootConfig:
@@ -83,7 +83,7 @@ def _validate(args: argparse.Namespace) -> int:
         print(f"  Severity: {e.policy.severity}")
     profile_arg = ""
     if args.profile:
-        profile_arg = f" -p {args.profile if args.profile.startswith('s3://') else os.path.realpath(args.profile)}"  # fmt: skip
+        profile_arg = f" -p {args.profile if args.profile.startswith(('s3://', 'env:')) else os.path.realpath(args.profile)}"  # fmt: skip
     print(f"Next: floe run -c {path}{profile_arg}")
     return 0
 

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/config/profile.py

@@ -1,5 +1,6 @@
 """Environment profiles, ported from `profile/parse.rs` and `profile/validate.rs`."""
 
+import os
 from collections.abc import Callable
 from dataclasses import dataclass
 from pathlib import Path
@@ -404,7 +405,20 @@ def apply_profile(config: RootConfig, profile: ProfileConfig) -> RootConfig:
     return config.model_copy(update={k: v for k, v in blocks.items() if v is not None})
 
 
+ENV_PREFIX = "env:"
+
+
+def read_env_profile(location: str) -> str:
+    """`env:NAME`: the profile YAML held in environment variable `NAME`."""
+    name = location.removeprefix(ENV_PREFIX)
+    if not (text := os.environ.get(name, "")).strip():
+        raise FloeConfigError(f"profile environment variable {name} is unset or empty")
+    return text
+
+
 def _docs(location: str | Path) -> list[object]:
+    if isinstance(location, str) and location.startswith(ENV_PREFIX):
+        return parse_yaml(read_env_profile(location))
     if isinstance(location, Path) or not location.startswith("s3://"):
         return load_yaml(Path(location))
     try:

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/iceberg/catalog.py

@@ -2,6 +2,9 @@
 
 import os
 import re
+import shutil
+import tempfile
+import weakref
 from collections.abc import Generator, Sequence
 from contextlib import contextmanager
 from urllib.parse import urlsplit
@@ -26,6 +29,11 @@ _TRANSFORMS = frozenset({"identity", "year", "month", "day", "hour"})
 # first id of a partition field, as in the Iceberg spec
 _PARTITION_FIELD_ID_START = 1000
 DUCKDB_CATALOG = "floe_catalog"
+# the ADR's limits (python-core-for-olf.md, "Supported limits"); DuckDB's own default is 80% of the
+# host's RAM, not the cgroup limit, so a Kubernetes pod is OOM-killed instead of spilling.
+# Size pods at the memory limit + 2 GiB. The spill bound fails a runaway merge before it fills the disk.
+DUCKDB_MEMORY_LIMIT = "4GB"
+DUCKDB_MAX_TEMP_DIRECTORY_SIZE = "50GB"
 
 
 def transform_name(field: PartitionField) -> str:
@@ -174,8 +182,10 @@ def open_duckdb(
         raise FloeConfigError(f"iceberg catalog {definition.name}: merges and compaction require a rest catalog (got {definition.type})")  # fmt: skip
     statements = _duckdb_setup(definition, storage)
     with run_error("duckdb catalog attach failed"):
-        # ponytail: DuckDB's default memory_limit and spill dir; set the ADR's limits when the run wires merges
-        con = duckdb.connect()
+        spill = tempfile.mkdtemp(prefix="floe-duckdb-")
+        con = duckdb.connect(config={"memory_limit": DUCKDB_MEMORY_LIMIT, "temp_directory": spill, "max_temp_directory_size": DUCKDB_MAX_TEMP_DIRECTORY_SIZE})  # fmt: skip
+        # the spill directory lives as long as the connection (also on `with` exit and at interpreter exit)
+        weakref.finalize(con, shutil.rmtree, spill, True)
         for statement in statements:
             con.execute(statement)
     return con

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/manifest.py

@@ -17,7 +17,7 @@ from typing import Any, NamedTuple, cast
 from pydantic import BaseModel
 
 from floe import __version__
-from floe.config.profile import ProfileConfig
+from floe.config.profile import ENV_PREFIX, ProfileConfig, read_env_profile
 from floe.config.project import ReportConfig, RootConfig
 from floe.config.storage import StoragesConfig
 from floe.config.types import ArchiveTarget, EntityConfig, SinkTarget, SourceConfig
@@ -464,7 +464,10 @@ def build_manifest(
     profile_uri = profile_checksum = None
     if options.profile_arg is not None:  # recorded like `config_uri` and its checksum
         # an s3 profile is read from there by the run, so it is recorded as typed
-        if options.profile_arg.startswith("s3://"):
+        if options.profile_arg.startswith(ENV_PREFIX):  # the Job reads the variable too
+            profile_uri = options.profile_arg
+            profile_checksum = _sha256(read_env_profile(profile_uri).encode())
+        elif options.profile_arg.startswith("s3://"):
             profile_uri = options.profile_arg
             profile_checksum = _sha256(read_s3(profile_uri, "profile"))
         else:

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/storage/local.py

@@ -101,6 +101,7 @@ class LocalClient:
         try:
             dst.parent.mkdir(parents=True, exist_ok=True)
             mode = src.stat().st_mode & 0o777
+            dst.unlink(missing_ok=True)  # not through a symlink or shared inode
             fd = os.open(dst, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode)
             os.fchmod(fd, mode)  # #779: the open mode applies on create only
             with src.open("rb") as fin, os.fdopen(fd, "wb") as fout:

{floe_python-0.8.2 → floe_python-0.8.3}/src/floe/write/outputs.py

@@ -4,10 +4,9 @@ import io
 import math
 import re
 import uuid
-from collections.abc import Generator, Iterator
-from contextlib import contextmanager
+from collections.abc import Callable, Generator
+from contextlib import contextmanager, suppress
 from dataclasses import dataclass
-from itertools import count
 
 import polars as pl
 
@@ -58,15 +57,46 @@ def part_uris(client: StorageClient, base: str, ext: str) -> list[str]:
     return sorted(u for u in uris if u.rpartition("/")[0] == base and u.endswith(f".{ext}") and _PART.fullmatch(u.rpartition("/")[2].removesuffix(f".{ext}")))  # fmt: skip
 
 
-def _part_names(client: StorageClient, base: str, mode: WriteMode, ext: str) -> Iterator[str]:  # fmt: skip
-    """Overwrite clears the old parts and numbers from 0 (`part-00000`); other modes use `part-<uuid>`."""
+@contextmanager
+def _parts(client: StorageClient, base: str, mode: WriteMode, ext: str) -> Generator[Callable[[bytes], str]]:  # fmt: skip
+    """Yield `put(body) -> name`. Non-overwrite modes write `part-<uuid>` directly.
+
+    Overwrite stages every part as `_staging-<id>-part-NNNNN` (invisible to `part_uris`) and
+    only when all were written copies them over `part-NNNNN`, deletes the old parts that no new
+    part replaced, then the staged copies (#1054). A failed write leaves the old parts untouched.
+    """
     if mode != "overwrite":
-        while True:
-            yield f"part-{uuid.uuid4()}.{ext}"
-    for uri in part_uris(client, base, ext):
+
+        def put(body: bytes) -> str:
+            name = f"part-{uuid.uuid4()}.{ext}"
+            client.write(f"{base}/{name}", body)
+            return name
+
+        yield put
+        return
+    stage = f"_staging-{uuid.uuid4()}-"
+    names: list[str] = []
+
+    def put_staged(body: bytes) -> str:
+        name = f"part-{len(names):05}.{ext}"
+        names.append(name)  # before the write: a partial file needs cleanup too
+        client.write(f"{base}/{stage}{name}", body)
+        return name
+
+    try:
+        yield put_staged
+    except BaseException:
+        for name in names:
+            with suppress(FloeError, OSError):  # best effort: the original error wins
+                client.delete(f"{base}/{stage}{name}")
+        raise
+    old = part_uris(client, base, ext)
+    for name in names:
+        client.copy(f"{base}/{stage}{name}", f"{base}/{name}")
+    for uri in set(old) - {f"{base}/{n}" for n in names}:
         client.delete(uri)
-    for i in count():
-        yield f"part-{i:05}.{ext}"
+    for name in names:
+        client.delete(f"{base}/{stage}{name}")
 
 
 def write_accepted(
@@ -78,7 +108,6 @@ def write_accepted(
 ) -> AcceptedWrite:
     """Write `df` as `part-*.parquet` files under `base`, split so each holds about `max_size_per_file`."""
     base = base.rstrip("/")
-    names = _part_names(client, base, mode, "parquet")
     max_size = (options and options.max_size_per_file) or DEFAULT_MAX_SIZE_PER_FILE
     avg_row = max(1, math.ceil(df.estimated_size() / df.height)) if df.height else 1
     rows = max(1, max_size // avg_row)
@@ -89,14 +118,12 @@ def write_accepted(
         kwargs["row_group_size"] = options.row_group_size
     files: list[str] = []
     sizes: list[int] = []
-    with _os_errors():
+    with _os_errors(), _parts(client, base, mode, "parquet") as put:
         for chunk in [df.slice(i, rows) for i in range(0, df.height, rows)] or [df]:
-            name = next(names)
             # ponytail: a part is held in memory (≤ max_size_per_file) before the put; stream multipart uploads if that is too much
             body = io.BytesIO()
             chunk.write_parquet(body, **kwargs)  # pyright: ignore[reportArgumentType]  # ponytail: kwargs loses the compression Literal; validated by the config
-            client.write(f"{base}/{name}", body.getvalue())
-            files.append(name)
+            files.append(put(body.getvalue()))
             sizes.append(body.tell())
     total = sum(sizes)
     return AcceptedWrite(
@@ -115,7 +142,10 @@ def write_rejected(df: pl.DataFrame, client: StorageClient, base: str, mode: Wri
     """
     with _os_errors():
         base = base.rstrip("/")
-        name = f"part-{uuid.uuid5(uuid.NAMESPACE_URL, key)}.csv" if key and mode != "overwrite" else next(_part_names(client, base, mode, "csv"))  # fmt: skip
-        uri = f"{base}/{name}"
-        client.write(uri, df.write_csv().encode())
-    return uri
+        if key and mode != "overwrite":
+            name = f"part-{uuid.uuid5(uuid.NAMESPACE_URL, key)}.csv"
+            client.write(f"{base}/{name}", df.write_csv().encode())
+        else:
+            with _parts(client, base, mode, "csv") as put:
+                name = put(df.write_csv().encode())
+    return f"{base}/{name}"