floe-python 0.8.1__tar.gz → 0.8.3__tar.gz

{floe_python-0.8.1 → floe_python-0.8.3}/CHANGELOG.md

@@ -2,13 +2,67 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.3
+
+- **`-p env:NAME` (#1068).** `-p` / `--profile` and `profile_path=` accept `env:NAME`, which reads the profile YAML
+  from environment variable `NAME`, so a `kubernetes_job` run needs no profile file in the Job (part of #1047).
+  An unset or empty variable is a `FloeConfigError` naming it. Relative paths a profile supplies resolve against
+  the config's directory, as for a file profile. `execution.runner` stays validate-only.
+- **A Parquet or CSV `overwrite` keeps the previous parts until the new ones are written (#1054)**: the
+  new parts are staged as `_staging-*` files, and the old parts are removed only after every new part
+  was written, so a failed write no longer empties the sink. A crash during the final swap can still
+  leave a mix of old and new parts; see [the Parquet sink](docs/sinks/parquet.md).
+- **Iceberg merges and compaction run DuckDB with `memory_limit=4GB` and a bounded spill directory (#1055).**
+  DuckDB defaulted to 80% of the host's RAM, so a large merge in a Kubernetes pod was OOM-killed
+  instead of spilling. It now caps memory at 4GB and spills to a temporary directory (at most
+  50GB, removed afterwards). Size pods at 6 GiB or more (see `docs/write_modes.md`).
+
 ## Unreleased
 
+### dagster-floe
+
+- **A `kubernetes_job` Job gets the profile as content, not as a pod-local path (#1069).** A run with a
+  profile reads it in the run pod and sets it as the Job's `FLOE_PROFILE_CONTENT` env var, with
+  `-p env:FLOE_PROFILE_CONTENT` in place of `-p <path>`; `${VAR}` references stay unexpanded and a remote
+  profile (`s3://...`) is passed as is. The Job image needs Floe 0.8.3 or newer for these runs.
+
+- **A `kubernetes_job` Job is deleted, with its pods, when the run stops watching it (#1070).** An
+  interrupt (a cancelled Dagster run), any error while polling, or the poll timeout used to leave the Job
+  writing tables, so a re-run could collide with it. A Job that reached a final state is left alone, so
+  `ttl_seconds_after_finished` still controls how long its logs stay.
+
 ### airflow-floe
 
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
   dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
+## dagster-floe v0.4.0
+
+**Upgrading from 0.3:** rejected assets move to new keys (below). Works with Floe 0.8.1 or newer; a Glue
+`database` with `{domain}` needs Floe 0.8.2.
+
+- **Breaking: rejected assets are keyed `<domain>/rejected/<table>` (#1046).** They were
+  `<domain>/<table>_rejected`, which collided with a real table named `<table>_rejected`. Upgrade:
+  the history of the rejected assets restarts under the new key, and any selection, sensor or
+  automation condition that names the old keys must be updated.
+
+- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
+  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
+
+- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
+  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
+
+- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
+  names the database the run writes.
+
+## v0.8.2
+
+- **A Glue catalog can write one database per domain (#1043).** `database` may contain `{domain}`,
+  e.g. `dev_{domain}_silver`: per entity it expands to `sink.accepted.iceberg.namespace`, else
+  `entity.domain`, so one catalog writes `dev_sales_silver` and `dev_finance_silver`. Put the catalog in
+  each stage's profile (`catalogs:` replaces the config's) to vary the prefix per stage. An entity
+  with neither is a `floe validate` error. A `database` without `{domain}` behaves as before.
+
 ## v0.8.1
 
 - **floe-python requires `boto3>=1.36.0` (was `>=1.43.118`) (#1028).** The old floor made `s3fs`

{floe_python-0.8.1 → floe_python-0.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.1
+Version: 0.8.3
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.1 → floe_python-0.8.3}/docs/cli.md

@@ -19,7 +19,8 @@
     `image` manifest or the current directory for a `cli` one. An `s3://` `report_base_uri` writes the
     reports there.
   - `-p` applies an environment profile (ignored with `--manifest`: the manifest is already compiled).
-    It takes a local path or an `s3://` URI, read as `--manifest` is (every command that takes `-p` does the same).
+    It takes a local path, an `s3://` URI read as `--manifest` is, or `env:NAME`, the profile YAML held in environment
+    variable `NAME` (every command that takes `-p` does the same).
   - `--domain` selects every entity of the named domains (repeatable or comma-separated), in config order.
     An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state

{floe_python-0.8.1 → floe_python-0.8.3}/docs/config.md

@@ -106,7 +106,8 @@ entities:
       - `warehouse_storage` (optional): storage name for table location resolution; must be an S3 storage
       - `warehouse_prefix` (optional): prefix for deterministic table locations
     - Glue definitions (`type: glue`) take `name`, `region` (required), `database` (required:
-      the Glue database that holds the tables), `create_database_if_missing` (optional, default
+      the Glue database that holds the tables; `{domain}` expands per entity to its
+      `sink.accepted.iceberg.namespace` or `domain`), `create_database_if_missing` (optional, default
       `true`), `warehouse_storage` (must be an S3 storage) and `warehouse_prefix`. Glue supports
       `overwrite` and `append`; merges need a REST catalog. See [Iceberg sink](sinks/iceberg.md).
 - `lineage` (optional)

{floe_python-0.8.1 → floe_python-0.8.3}/docs/profiles.md

@@ -51,7 +51,7 @@ All other fields are optional.
 
 ## Profile location
 
-A profile is a local path or an `s3://` URI, for `-p` / `--profile` on every command and for `profile_path=`
+A profile is a local path, an `s3://` URI or `env:NAME`, for `-p` / `--profile` on every command and for `profile_path=`
 in the Python API:
 
 ```bash
@@ -63,6 +63,19 @@ An `s3://` profile is read with the AWS default credential chain (`AWS_ENDPOINT_
 `profile not found: <uri>`. `floe manifest generate -p s3://...` records the URI as `profile_uri`, and the
 SHA-256 of the object as `profile_checksum`. The config (`-c`) is still a local path.
 
+`-p env:NAME` reads the profile YAML from environment variable `NAME`, so a container or Job needs no profile file:
+
+```bash
+FLOE_PROFILE_CONTENT="$(cat profile.yml)" floe run -c config.yml -p env:FLOE_PROFILE_CONTENT
+```
+
+An unset or empty variable fails with a `FloeConfigError` naming it. The profile behaves exactly as a file one
+(`${VAR}` references are expanded the same way). A profile never rebases paths, so a relative path it supplies
+(e.g. through `variables`) resolves against the config's directory, as for a file profile; there is no profile
+directory to resolve against. `execution.runner` is validated only: `floe run` always runs in-process and never launches
+a runner, so a Job started with its own profile does not start another Job. `floe manifest generate -p env:NAME`
+records `env:NAME` as `profile_uri` and the SHA-256 of the variable's content as `profile_checksum`.
+
 ---
 
 ## Variable precedence and resolution
@@ -303,7 +316,8 @@ Profiles can carry a `catalogs:` block with the same catalog definition shape as
 main Floe configs. When a command is run with both `--config` and `--profile`,
 the profile `catalogs:` block replaces the config-level `catalogs:` block for
 that invocation. This keeps entity logic in the shared config while moving
-REST Iceberg catalog endpoints (the only catalog type in 0.8) into environment-specific files.
+Iceberg catalog endpoints and Glue databases (e.g. `database: dev_{domain}_silver`) into
+environment-specific files.
 
 ```yaml
 apiVersion: floe/v1

{floe_python-0.8.1 → floe_python-0.8.3}/docs/sinks/iceberg.md

@@ -85,8 +85,14 @@ catalogs:
 ```
 
 The table is `<database>.<table>`; `sink.accepted.iceberg.namespace` (or `entity.domain`) only
-names its location, `<warehouse_prefix>/<namespace>/<table>`. Glue credentials come from the AWS
-credential chain, as for S3. Glue supports `overwrite` and `append`: `floe validate` rejects a
+names its location, `<warehouse_prefix>/<namespace>/<table>`. To write one database per domain from
+a single catalog, put `{domain}` in `database`, e.g. `database: dev_{domain}_silver`: per
+entity it becomes `sink.accepted.iceberg.namespace`, else `entity.domain`, normalized like the
+namespace (lowercase, `[a-z0-9_-]`), so the `sales` and `finance` domains write to `dev_sales_silver`
+and `dev_finance_silver`. An entity with neither a namespace nor a domain is a `floe validate`
+error. `{{VAR}}` placeholders are not applied to catalog definitions: to vary the prefix per stage,
+define the catalog in each stage's profile ([profile catalogs](../profiles.md#environment-catalog-overrides)
+replace the config's). Glue credentials come from the AWS credential chain, as for S3. Glue supports `overwrite` and `append`: `floe validate` rejects a
 merge write mode, and `floe maintain --compact` needs a REST catalog. `allow_takeover` is
 accepted and ignored: Floe 0.8 writes any Iceberg table in the database (D42).
 

{floe_python-0.8.1 → floe_python-0.8.3}/docs/sinks/parquet.md

@@ -18,7 +18,7 @@ entities:
 Semantics:
 - `sink.accepted.format: parquet` writes one or more part files under `sink.accepted.path`.
 - Write mode comes from `sink.write_mode`:
-  - `overwrite`: remove the existing `part-*.parquet` files, then write `part-00000.parquet`, `part-00001.parquet`, ...
+  - `overwrite`: write the new parts as `part-00000.parquet`, `part-00001.parquet`, ... and remove the old `part-*.parquet` files once every new part was written
   - `append`: add new `part-<uuid>.parquet` files alongside existing ones.
   - `merge_scd1` / `merge_scd2`: not supported for Parquet; use the [Iceberg](iceberg.md) sink.
 
@@ -60,10 +60,14 @@ with valid output files in the accepted directory.
 
 **Overwrite is not atomic**
 
-`overwrite` deletes the old part files before it writes the new ones. A failure in between
-leaves the directory empty or partly written until the next run. The run report and
-incremental state are not committed in that case, so Floe re-processes the same inputs on the
-next run.
+`overwrite` first writes every new part under a `_staging-<id>-part-NNNNN.parquet` name, which
+readers of `part-*.parquet` do not see. Only when all of them were written does it copy them to
+`part-NNNNN.parquet`, delete the old parts that were not replaced, then delete the staged files.
+A write failure leaves the previous parts untouched (the staged files are removed). A crash or a
+storage error during the final copy and delete steps can leave a mix of new and old parts (rows
+duplicated or missing) and orphaned `_staging-*` files, which are safe to delete. The run report
+and incremental state are not committed in that case, so Floe re-processes the same inputs on
+the next run and a re-run restores a clean directory.
 
 **If you need transactional accepted writes, use the [Iceberg](iceberg.md) sink instead.**
 Each entity's Iceberg write is one atomic catalog commit.

{floe_python-0.8.1 → floe_python-0.8.3}/docs/write_modes.md

@@ -54,6 +54,13 @@ row closes its current version (`valid_to` set, `is_current` false) and inserts
 unchanged rows are left alone. `compare_columns` and `ignore_columns` choose which columns count as a
 change. The report adds `closed_count` and `unchanged_count` to the merge metrics.
 
+### Sizing merge pods
+
+Merges and compaction run DuckDB with `memory_limit=4GB`, spilling to a temporary directory capped at
+`max_temp_directory_size=50GB` (removed when the run ends). DuckDB's resident memory overshoots its limit
+by 0.2-1.9 GiB, more as delete files accumulate, so size the pod at `memory_limit` + 2 GiB (6 GiB), and give
+it ephemeral disk for the spill.
+
 ## Rejected output behavior
 
 `sink.write_mode` is shared with rejected output:

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/README.md

@@ -42,10 +42,16 @@ pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.1,<0.9 for t
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |
 | 0.3.1 | 0.8.1 or newer (`job_scope: domain` Jobs run `floe run --domain`) | `0.8` |
 | 0.3.0 | 0.8 | `0.8` |
 | 0.2.x | 0.7 and earlier | see 0.2.x docs |
 
+With the `kubernetes_job` runner, a run that has a profile (config mode, `profile_path=`) reads the profile in the
+run pod and hands its text to the Job as the `FLOE_PROFILE_CONTENT` env var, with `-p env:FLOE_PROFILE_CONTENT`
+in place of the pod-local path. `${VAR}` references stay unexpanded: the Job expands them. The Job image needs
+Floe 0.8.3 or newer for these runs. A profile that is already a URI such as `s3://...` is passed as is.
+
 ## Development install (from this repo)
 
 ```bash
@@ -97,14 +103,15 @@ Dagster. The job name (`floe_<manifest_id>_job`) still derives from the manifest
 
 Entities of a source named in the manifest `sources` index share one `sources/<project>/<source>` dependency.
 Pass `source_key_fn` to `load_floe_assets` or `build_definitions` to give each entity its own upstream asset
-instead (for example, to depend on `crm/orders` and `crm/accounts` in domain mode):
+instead (for example, to depend on `crm/orders` and `crm/accounts` in domain mode). `entity.source_ref` and
+`entity.source_resource` carry the entity's `source.ref` and `source.resource` (`None` when absent):
 
 ```python
 from dagster import AssetKey
 
 defs = build_definitions(
     manifest_path="manifests/sales.json",
-    source_key_fn=lambda entity: AssetKey(["crm", entity.name]),  # None keeps the default key
+    source_key_fn=lambda e: AssetKey([e.source_ref, e.source_resource]) if e.source_resource else None,  # None: default key
 )
 ```
 
@@ -180,6 +187,10 @@ floe manifest generate -c orders.yml -p prod.yml --output manifests/orders.json
 
 See [docs/lineage.md](../../docs/lineage.md) for the full lineage config reference.
 
+## Output metadata
+
+Every accepted and rejected asset carries `floe/entity` (the manifest entity name) and `floe/output` (`"accepted"` or `"rejected"`) in its asset metadata, so tooling can tell the two apart without parsing dagster-floe's output names: `spec.metadata["floe/output"]` on each `AssetSpec` of `multi_asset.specs`.
+
 ## Notes
 
 - This connector does **not** parse YAML directly; it consumes Floe manifests (`spec_version` `0.8`), read from a file or built by `floe.build_manifest` in config mode.

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.3.1"
+version = "0.4.0"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -220,6 +220,11 @@ def _has_rejected_asset(entity: ManifestEntity) -> bool:
     return entity.rejected_sink_uri is not None
 
 
+def _out_metadata(entity: ManifestEntity, output: str) -> dict[str, str]:
+    """Lets callers tell accepted from rejected outputs without parsing dagster-floe's output names."""
+    return {"floe/entity": entity.name, "floe/output": output}
+
+
 def _safe_int(val: Any, default: int = 0) -> int:
     try:
         return int(val)
@@ -284,10 +289,14 @@ def _make_entity_multi_asset(
     entity_name = entity.name
 
     outs: dict[str, AssetOut] = {
-        "accepted": AssetOut(key=AssetKey(accepted_key), is_required=True),
+        "accepted": AssetOut(
+            key=AssetKey(accepted_key), is_required=True, metadata=_out_metadata(entity, "accepted")
+        ),
     }
     if has_rejected:
-        outs["rejected"] = AssetOut(key=AssetKey(rejected_key), is_required=False)
+        outs["rejected"] = AssetOut(
+            key=AssetKey(rejected_key), is_required=False, metadata=_out_metadata(entity, "rejected")
+        )
 
     @multi_asset(
         name=asset_name,
@@ -468,7 +477,8 @@ def _entity_events(
 
 
 def _rejected_key(entity: ManifestEntity) -> list[str]:
-    return list(entity.asset_key[:-1]) + [entity.asset_key[-1] + "_rejected"]
+    # A `rejected` segment, not a suffix: `orders_rejected` may be a real table next to `orders`.
+    return [*entity.asset_key[:-1], "rejected", entity.asset_key[-1]]
 
 
 # Each entity is reported from its own summary status: `rejected` and `success_with_warnings` entities
@@ -506,11 +516,17 @@ def _make_domain_multi_asset(
         op = _op_name(entity)
         # optional: a subsettable asset yields only its selected outputs, and fails the step itself
         outs[f"{op}__accepted"] = AssetOut(
-            key=AssetKey(entity.asset_key), group_name=entity.group_name, is_required=False
+            key=AssetKey(entity.asset_key),
+            group_name=entity.group_name,
+            is_required=False,
+            metadata=_out_metadata(entity, "accepted"),
         )
         if _has_rejected_asset(entity):
             outs[f"{op}__rejected"] = AssetOut(
-                key=AssetKey(_rejected_key(entity)), group_name=entity.group_name, is_required=False
+                key=AssetKey(_rejected_key(entity)),
+                group_name=entity.group_name,
+                is_required=False,
+                metadata=_out_metadata(entity, "rejected"),
             )
         check_specs.extend(build_asset_check_specs(list(entity.asset_key)))
     # each output keeps its own entity's source, so selecting one source's downstream runs one entity

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/src/floe_dagster/kubernetes_runner.py

@@ -1,8 +1,10 @@
 from __future__ import annotations
 
+import logging
 import time
 import uuid
 from dataclasses import replace
+from pathlib import Path
 from typing import Any
 
 from .k8s_status import (
@@ -18,6 +20,9 @@ from .k8s_status import (
 from .manifest import ManifestRunnerDefinition
 from .runner import RunResult
 
+_log = logging.getLogger(__name__)
+
+_PROFILE_ENV = "FLOE_PROFILE_CONTENT"
 _DEFAULT_POLL_INTERVAL_S = 10
 _DEFAULT_TIMEOUT_S = 3600
 
@@ -50,19 +55,26 @@ def run_kubernetes_job(
     deadline = time.monotonic() + timeout
     status = STATUS_SUBMITTED
     last_job: Any = None
+    reached_final_state = False
 
-    while True:
-        last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
-        status = map_k8s_job_status(last_job)
-        if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
-            break
-        if time.monotonic() >= deadline:
-            status = STATUS_TIMEOUT
-            break
-        time.sleep(poll_interval)
-
-    stdout = _collect_job_logs(core_api, job_name, namespace)
-    pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    try:
+        while True:
+            last_job = jobs_api.read_namespaced_job(name=job_name, namespace=namespace)
+            status = map_k8s_job_status(last_job)
+            if status in (STATUS_SUCCEEDED, STATUS_FAILED, STATUS_TIMEOUT, STATUS_CANCELED):
+                reached_final_state = True
+                break
+            if time.monotonic() >= deadline:
+                status = STATUS_TIMEOUT
+                break
+            time.sleep(poll_interval)
+        # before any deletion: with background propagation the pod would be gone
+        stdout = _collect_job_logs(core_api, job_name, namespace)
+        pod_ctx = _collect_pod_context(core_api, job_name, namespace)
+    finally:
+        # an interrupt, an error or the poll timeout leaves a Job nobody watches: it would keep writing
+        if not reached_final_state:
+            _delete_job(jobs_api, job_name, namespace)
 
     failure_reason = _resolve_failure_reason(status=status, job=last_job, pod_ctx=pod_ctx)
     if status == STATUS_SUCCEEDED:
@@ -92,6 +104,14 @@ def run_kubernetes_job(
     )
 
 
+def _delete_job(jobs_api: Any, job_name: str, namespace: str) -> None:
+    """Best effort, with its pods: a failure is logged and must not mask the error being raised."""
+    try:
+        jobs_api.delete_namespaced_job(job_name, namespace, propagation_policy="Background")
+    except Exception:
+        _log.warning("could not delete Kubernetes Job %s/%s", namespace, job_name, exc_info=True)
+
+
 def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunnerDefinition) -> dict[str, Any]:
     if not runner.image:
         raise ValueError("kubernetes_job runner requires a non-empty 'image' field")
@@ -102,6 +122,7 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     container_args: list[str] = list(cmd_args[1:]) if len(cmd_args) > 1 else []
     if runner.command is not None:
         container_cmd = runner.command
+    container_args, profile_text = _inline_profile(container_args)
     if runner.args is not None:  # a prefix: replacing the list would drop the entity selection
         container_args = runner.args + container_args
 
@@ -123,6 +144,9 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
                 }
             )
 
+    if profile_text is not None:
+        env_vars.append({"name": _PROFILE_ENV, "value": profile_text})
+
     container: dict[str, Any] = {"name": "floe-runner", "image": runner.image}
     if container_cmd:
         container["command"] = container_cmd
@@ -178,6 +202,17 @@ def build_k8s_job_spec(job_name: str, cmd_args: list[str], runner: ManifestRunne
     }
 
 
+def _inline_profile(args: list[str]) -> tuple[list[str], str | None]:
+    """A profile path only exists on the run pod's disk: the Job gets its text (`${VAR}` unexpanded) instead."""
+    for i, arg in enumerate(args[:-1]):
+        if arg in ("-p", "--profile"):
+            path = args[i + 1]
+            if "://" in path or path.startswith("env:"):  # already reachable from the Job
+                return args, None
+            return [*args[:i], "-p", f"env:{_PROFILE_ENV}", *args[i + 2 :]], Path(path).read_text(encoding="utf-8")
+    return args, None
+
+
 def _resource_requests(resources: dict[str, Any] | None) -> dict[str, str]:
     requests: dict[str, str] = {}
     if resources:

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/src/floe_dagster/manifest.py

@@ -31,6 +31,7 @@ class ManifestEntity:
     write_mode: str | None = None
     incremental_mode: str | None = None
     source_ref: str | None = None
+    source_resource: str | None = None
     iceberg: dict[str, Any] | None = None  # sinks.accepted.iceberg when the sink is Iceberg
 
     @staticmethod
@@ -56,6 +57,7 @@ class ManifestEntity:
             write_mode=_optional_str(data, "write_mode"),
             incremental_mode=_optional_str(data, "incremental_mode"),
             source_ref=_optional_str(source, "ref") if isinstance(source, dict) else None,
+            source_resource=_optional_str(source, "resource") if isinstance(source, dict) else None,
             iceberg=iceberg,
         )
 
@@ -473,7 +475,8 @@ def entity_table(manifest: DagsterManifest, entity: ManifestEntity) -> str | Non
     definitions = (manifest.catalogs or {}).get("definitions") or []
     config = next((d.get("type_config") or {} for d in definitions if d.get("name") == catalog), {})
     if config.get("type") == "glue":  # the run writes the Glue database; the namespace only names the location
-        namespace = config.get("database") or namespace
+        scope = entity.iceberg.get("namespace") or entity.domain or ""  # what `{domain}` stands for
+        namespace = (config.get("database") or namespace).replace("{domain}", _ident(scope))
     table = entity.iceberg.get("table") or entity.asset_key[-1]  # the run's unqualified name
     return ".".join(_ident(part) for part in (catalog, namespace, table))
 

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/tests/test_assets_manifest.py

@@ -146,14 +146,54 @@ def test_multi_asset_accepted_key_matches_entity_asset_key() -> None:
 def test_multi_asset_rejected_key_present_when_policy_is_reject() -> None:
     """employees has policy_severity='reject' → rejected output exposed."""
     asset_def, entity = _get_asset_def(FIXTURE, "employees")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key in asset_def.keys
 
 
+def test_rejected_key_is_a_segment_between_domain_and_table() -> None:
+    """[sales, orders] → [sales, rejected, orders]."""
+    from floe_dagster.assets import _rejected_key
+
+    _, entity = _get_asset_def(FIXTURE, "orders")
+    assert entity.asset_key == ["sales", "orders"]
+    assert _rejected_key(entity) == ["sales", "rejected", "orders"]
+
+
+def test_table_named_like_the_old_rejected_suffix_does_not_collide(tmp_path) -> None:
+    """A real `employees_rejected` table next to `employees` (which has a rejected sink)."""
+    import copy
+    import json
+
+    payload = json.loads(FIXTURE.read_text(encoding="utf-8"))
+    employees = payload["entities"][0]
+    sibling = copy.deepcopy(employees)
+    sibling["name"] = "employees_rejected"
+    sibling["asset_key"] = ["hr", "employees_rejected"]
+    payload["entities"].append(sibling)
+    manifest_path = tmp_path / "manifest.sibling.json"
+    manifest_path.write_text(json.dumps(payload), encoding="utf-8")
+
+    defs = build_definitions(manifest_path=str(manifest_path), runner=_NoopRunner(), with_job=False)
+    defs.resolve_implicit_global_asset_job_def()  # raises on a duplicate asset key
+    keys = {key for asset in defs.assets if hasattr(asset, "keys") for key in asset.keys}
+    assert AssetKey(["hr", "employees_rejected"]) in keys
+    assert AssetKey(["hr", "rejected", "employees"]) in keys
+    assert AssetKey(["hr", "rejected", "employees_rejected"]) in keys
+
+
+def test_multi_asset_outputs_carry_entity_and_output_metadata() -> None:
+    asset_def, entity = _get_asset_def(FIXTURE, "employees")
+    by_output = {spec.metadata["floe/output"]: spec for spec in asset_def.specs}
+    assert set(by_output) == {"accepted", "rejected"}
+    assert all(spec.metadata["floe/entity"] == entity.name for spec in by_output.values())
+    assert by_output["accepted"].key == AssetKey(entity.asset_key)
+    assert by_output["rejected"].key == AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
+
+
 def test_multi_asset_no_rejected_key_when_policy_is_warn() -> None:
     """orders has policy_severity='warn' → no rejected output (rows discarded silently)."""
     asset_def, entity = _get_asset_def(FIXTURE, "orders")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key not in asset_def.keys
 
 
@@ -169,7 +209,7 @@ def test_multi_asset_rejected_key_present_when_policy_is_abort(tmp_path) -> None
     manifest_path.write_text(json.dumps(payload), encoding="utf-8")
 
     asset_def, entity = _get_asset_def(manifest_path, "employees")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key in asset_def.keys
 
 

{floe_python-0.8.1 → floe_python-0.8.3}/orchestrators/dagster-floe/tests/test_domain_scope.py

@@ -126,6 +126,23 @@ def test_domain_scope_failed_entity_fails_alone_and_the_ones_before_it_materiali
     assert "entities returns; not run: refunds" in message
 
 
+def test_domain_scope_outputs_carry_entity_and_output_metadata(tmp_path: Path) -> None:
+    manifest = _manifest(tmp_path)
+    payload = json.loads(manifest.read_text(encoding="utf-8"))
+    for entity in payload["entities"]:
+        entity["policy_severity"] = "reject"
+    manifest.write_text(json.dumps(payload), encoding="utf-8")
+
+    assets, _, _ = build_floe_asset_defs(manifest_path=str(manifest), runner=LocalRunner())
+    [domain_asset] = assets
+    seen = {(spec.metadata["floe/entity"], spec.metadata["floe/output"]): spec.key for spec in domain_asset.specs}
+    assert seen == {
+        (name, output): AssetKey(["sales", *segment, name])
+        for name in NAMES
+        for output, segment in (("accepted", []), ("rejected", ["rejected"]))
+    }
+
+
 def test_entity_scope_is_unchanged_when_job_scope_is_unset(tmp_path: Path) -> None:
     result, jobs_api = _run(tmp_path, {"orders": "success"}, job_scope=None, selection=[AssetKey(["sales", "orders"])])