floe-python 0.8.1__tar.gz → 0.8.2__tar.gz

{floe_python-0.8.1 → floe_python-0.8.2}/CHANGELOG.md

@@ -4,11 +4,35 @@ All notable changes to Floe are documented in this file.
 
 ## Unreleased
 
+### dagster-floe
+
+- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
+  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
+
+- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
+  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
+
+- **Breaking: rejected assets are keyed `<domain>/rejected/<table>` (#1046).** They were
+  `<domain>/<table>_rejected`, which collided with a real table named `<table>_rejected`. Upgrade:
+  the history of the rejected assets restarts under the new key, and any selection, sensor or
+  automation condition that names the old keys must be updated.
+
+- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
+  names the database the run writes.
+
 ### airflow-floe
 
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
   dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
+## v0.8.2
+
+- **A Glue catalog can write one database per domain (#1043).** `database` may contain `{domain}`,
+  e.g. `dev_{domain}_silver`: per entity it expands to `sink.accepted.iceberg.namespace`, else
+  `entity.domain`, so one catalog writes `dev_sales_silver` and `dev_finance_silver`. Put the catalog in
+  each stage's profile (`catalogs:` replaces the config's) to vary the prefix per stage. An entity
+  with neither is a `floe validate` error. A `database` without `{domain}` behaves as before.
+
 ## v0.8.1
 
 - **floe-python requires `boto3>=1.36.0` (was `>=1.43.118`) (#1028).** The old floor made `s3fs`

{floe_python-0.8.1 → floe_python-0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.1
+Version: 0.8.2
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.1 → floe_python-0.8.2}/docs/config.md

@@ -106,7 +106,8 @@ entities:
       - `warehouse_storage` (optional): storage name for table location resolution; must be an S3 storage
       - `warehouse_prefix` (optional): prefix for deterministic table locations
     - Glue definitions (`type: glue`) take `name`, `region` (required), `database` (required:
-      the Glue database that holds the tables), `create_database_if_missing` (optional, default
+      the Glue database that holds the tables; `{domain}` expands per entity to its
+      `sink.accepted.iceberg.namespace` or `domain`), `create_database_if_missing` (optional, default
       `true`), `warehouse_storage` (must be an S3 storage) and `warehouse_prefix`. Glue supports
       `overwrite` and `append`; merges need a REST catalog. See [Iceberg sink](sinks/iceberg.md).
 - `lineage` (optional)

{floe_python-0.8.1 → floe_python-0.8.2}/docs/profiles.md

@@ -303,7 +303,8 @@ Profiles can carry a `catalogs:` block with the same catalog definition shape as
 main Floe configs. When a command is run with both `--config` and `--profile`,
 the profile `catalogs:` block replaces the config-level `catalogs:` block for
 that invocation. This keeps entity logic in the shared config while moving
-REST Iceberg catalog endpoints (the only catalog type in 0.8) into environment-specific files.
+Iceberg catalog endpoints and Glue databases (e.g. `database: dev_{domain}_silver`) into
+environment-specific files.
 
 ```yaml
 apiVersion: floe/v1

{floe_python-0.8.1 → floe_python-0.8.2}/docs/sinks/iceberg.md

@@ -85,8 +85,14 @@ catalogs:
 ```
 
 The table is `<database>.<table>`; `sink.accepted.iceberg.namespace` (or `entity.domain`) only
-names its location, `<warehouse_prefix>/<namespace>/<table>`. Glue credentials come from the AWS
-credential chain, as for S3. Glue supports `overwrite` and `append`: `floe validate` rejects a
+names its location, `<warehouse_prefix>/<namespace>/<table>`. To write one database per domain from
+a single catalog, put `{domain}` in `database`, e.g. `database: dev_{domain}_silver`: per
+entity it becomes `sink.accepted.iceberg.namespace`, else `entity.domain`, normalized like the
+namespace (lowercase, `[a-z0-9_-]`), so the `sales` and `finance` domains write to `dev_sales_silver`
+and `dev_finance_silver`. An entity with neither a namespace nor a domain is a `floe validate`
+error. `{{VAR}}` placeholders are not applied to catalog definitions: to vary the prefix per stage,
+define the catalog in each stage's profile ([profile catalogs](../profiles.md#environment-catalog-overrides)
+replace the config's). Glue credentials come from the AWS credential chain, as for S3. Glue supports `overwrite` and `append`: `floe validate` rejects a
 merge write mode, and `floe maintain --compact` needs a REST catalog. `allow_takeover` is
 accepted and ignored: Floe 0.8 writes any Iceberg table in the database (D42).
 

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/README.md

@@ -97,14 +97,15 @@ Dagster. The job name (`floe_<manifest_id>_job`) still derives from the manifest
 
 Entities of a source named in the manifest `sources` index share one `sources/<project>/<source>` dependency.
 Pass `source_key_fn` to `load_floe_assets` or `build_definitions` to give each entity its own upstream asset
-instead (for example, to depend on `crm/orders` and `crm/accounts` in domain mode):
+instead (for example, to depend on `crm/orders` and `crm/accounts` in domain mode). `entity.source_ref` and
+`entity.source_resource` carry the entity's `source.ref` and `source.resource` (`None` when absent):
 
 ```python
 from dagster import AssetKey
 
 defs = build_definitions(
     manifest_path="manifests/sales.json",
-    source_key_fn=lambda entity: AssetKey(["crm", entity.name]),  # None keeps the default key
+    source_key_fn=lambda e: AssetKey([e.source_ref, e.source_resource]) if e.source_resource else None,  # None: default key
 )
 ```
 
@@ -180,6 +181,10 @@ floe manifest generate -c orders.yml -p prod.yml --output manifests/orders.json
 
 See [docs/lineage.md](../../docs/lineage.md) for the full lineage config reference.
 
+## Output metadata
+
+Every accepted and rejected asset carries `floe/entity` (the manifest entity name) and `floe/output` (`"accepted"` or `"rejected"`) in its asset metadata, so tooling can tell the two apart without parsing dagster-floe's output names: `spec.metadata["floe/output"]` on each `AssetSpec` of `multi_asset.specs`.
+
 ## Notes
 
 - This connector does **not** parse YAML directly; it consumes Floe manifests (`spec_version` `0.8`), read from a file or built by `floe.build_manifest` in config mode.

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -220,6 +220,11 @@ def _has_rejected_asset(entity: ManifestEntity) -> bool:
     return entity.rejected_sink_uri is not None
 
 
+def _out_metadata(entity: ManifestEntity, output: str) -> dict[str, str]:
+    """Lets callers tell accepted from rejected outputs without parsing dagster-floe's output names."""
+    return {"floe/entity": entity.name, "floe/output": output}
+
+
 def _safe_int(val: Any, default: int = 0) -> int:
     try:
         return int(val)
@@ -284,10 +289,14 @@ def _make_entity_multi_asset(
     entity_name = entity.name
 
     outs: dict[str, AssetOut] = {
-        "accepted": AssetOut(key=AssetKey(accepted_key), is_required=True),
+        "accepted": AssetOut(
+            key=AssetKey(accepted_key), is_required=True, metadata=_out_metadata(entity, "accepted")
+        ),
     }
     if has_rejected:
-        outs["rejected"] = AssetOut(key=AssetKey(rejected_key), is_required=False)
+        outs["rejected"] = AssetOut(
+            key=AssetKey(rejected_key), is_required=False, metadata=_out_metadata(entity, "rejected")
+        )
 
     @multi_asset(
         name=asset_name,
@@ -468,7 +477,8 @@ def _entity_events(
 
 
 def _rejected_key(entity: ManifestEntity) -> list[str]:
-    return list(entity.asset_key[:-1]) + [entity.asset_key[-1] + "_rejected"]
+    # A `rejected` segment, not a suffix: `orders_rejected` may be a real table next to `orders`.
+    return [*entity.asset_key[:-1], "rejected", entity.asset_key[-1]]
 
 
 # Each entity is reported from its own summary status: `rejected` and `success_with_warnings` entities
@@ -506,11 +516,17 @@ def _make_domain_multi_asset(
         op = _op_name(entity)
         # optional: a subsettable asset yields only its selected outputs, and fails the step itself
         outs[f"{op}__accepted"] = AssetOut(
-            key=AssetKey(entity.asset_key), group_name=entity.group_name, is_required=False
+            key=AssetKey(entity.asset_key),
+            group_name=entity.group_name,
+            is_required=False,
+            metadata=_out_metadata(entity, "accepted"),
         )
         if _has_rejected_asset(entity):
             outs[f"{op}__rejected"] = AssetOut(
-                key=AssetKey(_rejected_key(entity)), group_name=entity.group_name, is_required=False
+                key=AssetKey(_rejected_key(entity)),
+                group_name=entity.group_name,
+                is_required=False,
+                metadata=_out_metadata(entity, "rejected"),
             )
         check_specs.extend(build_asset_check_specs(list(entity.asset_key)))
     # each output keeps its own entity's source, so selecting one source's downstream runs one entity

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/src/floe_dagster/manifest.py

@@ -31,6 +31,7 @@ class ManifestEntity:
     write_mode: str | None = None
     incremental_mode: str | None = None
     source_ref: str | None = None
+    source_resource: str | None = None
     iceberg: dict[str, Any] | None = None  # sinks.accepted.iceberg when the sink is Iceberg
 
     @staticmethod
@@ -56,6 +57,7 @@ class ManifestEntity:
             write_mode=_optional_str(data, "write_mode"),
             incremental_mode=_optional_str(data, "incremental_mode"),
             source_ref=_optional_str(source, "ref") if isinstance(source, dict) else None,
+            source_resource=_optional_str(source, "resource") if isinstance(source, dict) else None,
             iceberg=iceberg,
         )
 
@@ -473,7 +475,8 @@ def entity_table(manifest: DagsterManifest, entity: ManifestEntity) -> str | Non
     definitions = (manifest.catalogs or {}).get("definitions") or []
     config = next((d.get("type_config") or {} for d in definitions if d.get("name") == catalog), {})
     if config.get("type") == "glue":  # the run writes the Glue database; the namespace only names the location
-        namespace = config.get("database") or namespace
+        scope = entity.iceberg.get("namespace") or entity.domain or ""  # what `{domain}` stands for
+        namespace = (config.get("database") or namespace).replace("{domain}", _ident(scope))
     table = entity.iceberg.get("table") or entity.asset_key[-1]  # the run's unqualified name
     return ".".join(_ident(part) for part in (catalog, namespace, table))
 

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/tests/test_assets_manifest.py

@@ -146,14 +146,54 @@ def test_multi_asset_accepted_key_matches_entity_asset_key() -> None:
 def test_multi_asset_rejected_key_present_when_policy_is_reject() -> None:
     """employees has policy_severity='reject' → rejected output exposed."""
     asset_def, entity = _get_asset_def(FIXTURE, "employees")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key in asset_def.keys
 
 
+def test_rejected_key_is_a_segment_between_domain_and_table() -> None:
+    """[sales, orders] → [sales, rejected, orders]."""
+    from floe_dagster.assets import _rejected_key
+
+    _, entity = _get_asset_def(FIXTURE, "orders")
+    assert entity.asset_key == ["sales", "orders"]
+    assert _rejected_key(entity) == ["sales", "rejected", "orders"]
+
+
+def test_table_named_like_the_old_rejected_suffix_does_not_collide(tmp_path) -> None:
+    """A real `employees_rejected` table next to `employees` (which has a rejected sink)."""
+    import copy
+    import json
+
+    payload = json.loads(FIXTURE.read_text(encoding="utf-8"))
+    employees = payload["entities"][0]
+    sibling = copy.deepcopy(employees)
+    sibling["name"] = "employees_rejected"
+    sibling["asset_key"] = ["hr", "employees_rejected"]
+    payload["entities"].append(sibling)
+    manifest_path = tmp_path / "manifest.sibling.json"
+    manifest_path.write_text(json.dumps(payload), encoding="utf-8")
+
+    defs = build_definitions(manifest_path=str(manifest_path), runner=_NoopRunner(), with_job=False)
+    defs.resolve_implicit_global_asset_job_def()  # raises on a duplicate asset key
+    keys = {key for asset in defs.assets if hasattr(asset, "keys") for key in asset.keys}
+    assert AssetKey(["hr", "employees_rejected"]) in keys
+    assert AssetKey(["hr", "rejected", "employees"]) in keys
+    assert AssetKey(["hr", "rejected", "employees_rejected"]) in keys
+
+
+def test_multi_asset_outputs_carry_entity_and_output_metadata() -> None:
+    asset_def, entity = _get_asset_def(FIXTURE, "employees")
+    by_output = {spec.metadata["floe/output"]: spec for spec in asset_def.specs}
+    assert set(by_output) == {"accepted", "rejected"}
+    assert all(spec.metadata["floe/entity"] == entity.name for spec in by_output.values())
+    assert by_output["accepted"].key == AssetKey(entity.asset_key)
+    assert by_output["rejected"].key == AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
+
+
 def test_multi_asset_no_rejected_key_when_policy_is_warn() -> None:
     """orders has policy_severity='warn' → no rejected output (rows discarded silently)."""
     asset_def, entity = _get_asset_def(FIXTURE, "orders")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key not in asset_def.keys
 
 
@@ -169,7 +209,7 @@ def test_multi_asset_rejected_key_present_when_policy_is_abort(tmp_path) -> None
     manifest_path.write_text(json.dumps(payload), encoding="utf-8")
 
     asset_def, entity = _get_asset_def(manifest_path, "employees")
-    rejected_key = AssetKey(entity.asset_key[:-1] + [entity.asset_key[-1] + "_rejected"])
+    rejected_key = AssetKey([*entity.asset_key[:-1], "rejected", entity.asset_key[-1]])
     assert rejected_key in asset_def.keys
 
 

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/tests/test_domain_scope.py

@@ -126,6 +126,23 @@ def test_domain_scope_failed_entity_fails_alone_and_the_ones_before_it_materiali
     assert "entities returns; not run: refunds" in message
 
 
+def test_domain_scope_outputs_carry_entity_and_output_metadata(tmp_path: Path) -> None:
+    manifest = _manifest(tmp_path)
+    payload = json.loads(manifest.read_text(encoding="utf-8"))
+    for entity in payload["entities"]:
+        entity["policy_severity"] = "reject"
+    manifest.write_text(json.dumps(payload), encoding="utf-8")
+
+    assets, _, _ = build_floe_asset_defs(manifest_path=str(manifest), runner=LocalRunner())
+    [domain_asset] = assets
+    seen = {(spec.metadata["floe/entity"], spec.metadata["floe/output"]): spec.key for spec in domain_asset.specs}
+    assert seen == {
+        (name, output): AssetKey(["sales", *segment, name])
+        for name in NAMES
+        for output, segment in (("accepted", []), ("rejected", ["rejected"]))
+    }
+
+
 def test_entity_scope_is_unchanged_when_job_scope_is_unset(tmp_path: Path) -> None:
     result, jobs_api = _run(tmp_path, {"orders": "success"}, job_scope=None, selection=[AssetKey(["sales", "orders"])])
 

{floe_python-0.8.1 → floe_python-0.8.2}/orchestrators/dagster-floe/tests/test_manifest.py

@@ -1,10 +1,12 @@
 import json
+from dataclasses import replace
 from pathlib import Path
 
 import pytest
 
 from floe_dagster.manifest import (
     DagsterManifest,
+    entity_table,
     load_manifest,
     render_execution_args,
     resolve_config_uri,
@@ -197,6 +199,33 @@ def test_entity_policy_severity_defaults_to_none_when_absent():
     assert manifest.entities[0].policy_severity is None
 
 
+def test_entity_table_expands_the_domain_token_of_a_glue_database():
+    fixture = Path(__file__).parent / "fixtures" / "manifest.json"
+    payload = json.loads(fixture.read_text(encoding="utf-8"))
+    glue = {"name": "glue", "type_config": {"type": "glue", "region": "eu-west-1", "database": "dev_{domain}_silver"}}
+    payload["catalogs"] = {"default": "glue", "definitions": [glue]}
+    manifest = DagsterManifest.from_dict(payload)
+    entity = manifest.entities[0]
+
+    def table(domain: str | None, namespace: str | None) -> str | None:
+        iceberg = {"table": "orders", **({"namespace": namespace} if namespace else {})}
+        return entity_table(manifest, replace(entity, domain=domain, iceberg=iceberg))
+
+    assert table("Sales", None) == "glue.dev_sales_silver.orders"
+    assert table("sales", "Finance EU") == "glue.dev_finance_eu_silver.orders"
+
+
+def test_entity_source_resource_parsed_and_defaults_to_none():
+    fixture = Path(__file__).parent / "fixtures" / "manifest.json"
+    payload = json.loads(fixture.read_text(encoding="utf-8"))
+    payload["entities"][0]["source"].update({"ref": "crm", "resource": "orders"})
+    payload["entities"][1]["source"].pop("resource", None)
+
+    entities = DagsterManifest.from_dict(payload).entities
+    assert (entities[0].source_ref, entities[0].source_resource) == ("crm", "orders")
+    assert entities[1].source_resource is None
+
+
 def test_manifest_schema_accepts_databricks_runner_fields(tmp_path: Path):
     fixture = Path(__file__).parent / "fixtures" / "manifest.json"
     payload = json.loads(fixture.read_text(encoding="utf-8"))

{floe_python-0.8.1 → floe_python-0.8.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.1"
+version = "0.8.2"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.1 → floe_python-0.8.2}/schemas/profile.schema.yaml

@@ -445,7 +445,7 @@ $defs:
         description: Glue catalog region; required when type is glue.
       database:
         type: string
-        description: Glue database; required when type is glue.
+        description: Glue database; required when type is glue. "{domain}" expands per entity to its iceberg namespace or domain.
       create_database_if_missing:
         type: boolean
       allow_takeover:

{floe_python-0.8.1 → floe_python-0.8.2}/schemas/project.schema.yaml

@@ -2088,7 +2088,7 @@ $defs:
         description: Glue catalog region; required when type is glue.
       database:
         type: string
-        description: Glue database; required when type is glue.
+        description: Glue database; required when type is glue. "{domain}" expands per entity to its iceberg namespace or domain.
       create_database_if_missing:
         type:
         - boolean

{floe_python-0.8.1 → floe_python-0.8.2}/src/floe/config/validate.py

@@ -330,6 +330,8 @@ def _check_iceberg(
         raise _fail(
             entity, f"sink.accepted.iceberg.catalog references unknown catalog {name}"
         )
+    if definition.type == "glue" and "{domain}" in (definition.database or "") and not (iceberg.namespace or entity.domain):  # fmt: skip
+        raise _fail(entity, f"sink.accepted.iceberg.namespace or domain is required for catalog {name} database containing {{domain}}")  # fmt: skip
     # Unity is rejected at parse (D1); the registry checked `warehouse_storage`.
     if definition.type == "glue" and entity.sink.write_mode.startswith("merge"):
         # ponytail: merges run through DuckDB, attached to REST catalogs only; attach Glue when a merge needs it

{floe_python-0.8.1 → floe_python-0.8.2}/src/floe/iceberg/sink.py

@@ -70,9 +70,12 @@ def resolve_target(storages: StoragesConfig | None, catalogs: CatalogsConfig | N
     catalog = next((c for c in (catalogs.definitions if catalogs else []) if c.name == name), None)  # fmt: skip
     if catalog is None:  # validated; kept for callers that skip validation
         raise FloeConfigError(f"entity.name={entity.name} sink.accepted.iceberg.catalog references unknown catalog {name}")  # fmt: skip
-    namespace = catalog_ident(
-        sink.iceberg.namespace or entity.domain or catalog.database or "default"
-    )
+    # what `{domain}` in the database stands for (validated: set when the database has the token)
+    scope = sink.iceberg.namespace or entity.domain
+    database = catalog.database
+    if database is not None:
+        database = database.replace("{domain}", catalog_ident(scope or ""))
+    namespace = catalog_ident(scope or database or "default")
     table = catalog_ident(sink.iceberg.table or entity.name)
     field = f"entity.name={entity.name} sink.accepted.storage"
     accepted_uri, storage = resolve_path(storages, config_dir, sink.storage, sink.path, field)  # fmt: skip
@@ -85,8 +88,8 @@ def resolve_target(storages: StoragesConfig | None, catalogs: CatalogsConfig | N
     else:
         location = accepted_uri
     # Rust: the Glue database holds the table; the namespace only names its location
-    if catalog.database is not None:
-        namespace = catalog.database
+    if database is not None:
+        namespace = database
     return IcebergTarget(catalog, storage, namespace, table, location, accepted_uri)  # fmt: skip
 
 

{floe_python-0.8.1 → floe_python-0.8.2}/tests/integration/test_iceberg_glue.py

@@ -80,3 +80,66 @@ def test_overwrite_then_append_lands_in_the_glue_database(
     table = catalog.load_table((database, "orders"))
     assert table.location() == f"s3://{BUCKET}/{prefix}/sales/orders"
     assert sorted(table.scan().to_arrow()["id"].to_pylist()) == [1, 2, 3, 4, 5]
+
+
+DOMAINS_CONFIG = """\
+version: "0.3"
+storages:
+  default: local
+  definitions:
+    - {{name: local, type: local}}
+    - {{name: lake, type: s3, bucket: {bucket}, region: us-east-1, endpoint: "{s3}", path_style_access: true}}
+catalogs:
+  default: glue
+  definitions:
+    - {{name: glue, type: glue, region: us-east-1, database: "{database}_{{domain}}", warehouse_storage: lake, warehouse_prefix: "{prefix}"}}
+domains:
+  - name: sales
+  - name: finance
+entities:
+  - name: orders
+    domain: sales
+    source: {{format: csv, path: {orders}, options: {{header: true}}}}
+    sink:
+      accepted: {{format: iceberg, path: orders, storage: lake, iceberg: {{table: orders}}}}
+    policy: {{severity: warn}}
+    schema:
+      columns:
+        - {{name: id, type: int64}}
+  - name: ledger
+    domain: finance
+    source: {{format: csv, path: {ledger}, options: {{header: true}}}}
+    sink:
+      accepted: {{format: iceberg, path: ledger, storage: lake, iceberg: {{table: ledger}}}}
+    policy: {{severity: warn}}
+    schema:
+      columns:
+        - {{name: id, type: int64}}
+"""
+
+
+def test_domain_token_writes_each_domain_to_its_own_glue_database(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    monkeypatch.setenv("AWS_ACCESS_KEY_ID", os.environ.get("AWS_ACCESS_KEY_ID", "test"))
+    monkeypatch.setenv(
+        "AWS_SECRET_ACCESS_KEY", os.environ.get("AWS_SECRET_ACCESS_KEY", "test")
+    )
+    monkeypatch.setenv("AWS_ENDPOINT_URL_GLUE", GLUE or "")
+    database, prefix = f"floe_{uuid.uuid4().hex[:8]}", f"glue-it/{uuid.uuid4().hex[:8]}"
+    s3 = boto3.client("s3", endpoint_url=S3, region_name="us-east-1")
+    try:
+        s3.create_bucket(Bucket=BUCKET)
+    except s3.exceptions.BucketAlreadyOwnedByYou:
+        pass
+    (tmp_path / "orders.csv").write_text("id\n1\n2\n")
+    (tmp_path / "ledger.csv").write_text("id\n7\n")
+    config = tmp_path / "floe.yml"
+    config.write_text(DOMAINS_CONFIG.format(bucket=BUCKET, s3=S3, database=database, prefix=prefix, orders=tmp_path / "orders.csv", ledger=tmp_path / "ledger.csv"))  # fmt: skip
+    run(config, parse_config(config), run_id="r-domains")
+
+    catalog = load_catalog("glue", type="glue", **{"glue.region": "us-east-1", "s3.endpoint": S3 or "", "s3.region": "us-east-1"})  # fmt: skip
+    for domain, table, ids in (("sales", "orders", [1, 2]), ("finance", "ledger", [7])):
+        loaded = catalog.load_table((f"{database}_{domain}", table))
+        assert loaded.location() == f"s3://{BUCKET}/{prefix}/{domain}/{table}"
+        assert sorted(loaded.scan().to_arrow()["id"].to_pylist()) == ids

{floe_python-0.8.1 → floe_python-0.8.2}/tests/unit/iceberg/test_glue.py

@@ -67,6 +67,73 @@ def test_glue_target_uses_the_database_and_the_warehouse_prefix(
     assert target.location == "s3://silver-bucket/warehouse/iceberg/sales/orders"
 
 
+ENTITY = """\
+  - name: {name}
+    {domain}
+    source: {{format: csv, path: in/{name}.csv, storage: silver}}
+    sink:
+      accepted:
+        format: iceberg
+        path: {name}
+        iceberg: {{catalog: glue{namespace}}}
+    policy: {{severity: warn}}
+    schema:
+      primary_key: [id]
+      columns:
+        - {{name: id, type: string, nullable: false}}
+"""
+
+
+def per_domain_config(tmp_path: Path, database: str, *entities: tuple[str, str | None, str | None]) -> Path:  # fmt: skip
+    """`entities` are (name, domain, namespace); the Glue `database` is the given template."""
+    head = CONFIG.split("entities:")[0].format(mode="").replace("database: sales_silver", f"database: {database!r}")  # fmt: skip
+    body = "".join(
+        ENTITY.format(name=n, domain=f"domain: {d}" if d else "", namespace=f", namespace: {ns}" if ns else "")
+        for n, d, ns in entities
+    )  # fmt: skip
+    path = tmp_path / "floe.yml"
+    path.write_text(
+        head
+        + "domains:\n  - {name: sales, incoming_dir: in/sales}\n  - {name: finance, incoming_dir: in/finance}\nentities:\n"
+        + body
+    )
+    return path
+
+
+def targets(path: Path) -> list[tuple[str, str]]:
+    root = parse_config(path)
+    validate_config(root)
+    found = [resolve_target(root.storages, root.catalogs, path.parent, e) for e in root.entities]  # fmt: skip
+    return [(t.namespace, t.location) for t in found]
+
+
+def test_glue_database_domain_token_expands_per_entity(tmp_path: Path) -> None:
+    path = per_domain_config(tmp_path, "dev_{domain}", ("orders", "sales", None), ("ledger", "finance", None))  # fmt: skip
+    assert targets(path) == [
+        ("dev_sales", "s3://silver-bucket/warehouse/iceberg/sales/orders"),
+        ("dev_finance", "s3://silver-bucket/warehouse/iceberg/finance/ledger"),
+    ]  # the location keeps the plain namespace, never the template
+
+
+def test_glue_database_domain_token_uses_the_namespace_over_the_domain(tmp_path: Path) -> None:  # fmt: skip
+    path = per_domain_config(tmp_path, "dev_{domain}_silver", ("orders", "sales", "Sales EU"))  # fmt: skip
+    assert targets(path) == [("dev_sales_eu_silver", "s3://silver-bucket/warehouse/iceberg/sales_eu/orders")]  # fmt: skip
+
+
+def test_glue_database_without_the_token_is_unchanged(tmp_path: Path) -> None:
+    path = per_domain_config(tmp_path, "fixed", ("orders", "sales", None), ("ledger", "finance", None))  # fmt: skip
+    assert [db for db, _ in targets(path)] == ["fixed", "fixed"]
+
+
+def test_glue_database_domain_token_needs_a_namespace_or_domain(tmp_path: Path) -> None:  # fmt: skip
+    path = per_domain_config(tmp_path, "dev_{domain}", ("orders", None, None))
+    with pytest.raises(FloeConfigError) as err:
+        validate_config(parse_config(path))
+    assert str(err.value) == (
+        "entity.name=orders sink.accepted.iceberg.namespace or domain is required for catalog glue database containing {domain}"
+    )
+
+
 def test_glue_catalog_opens_through_pyiceberg(tmp_path: Path) -> None:
     root = parse_config(config(tmp_path))
     assert root.catalogs is not None

{floe_python-0.8.1 → floe_python-0.8.2}/uv.lock

@@ -450,7 +450,7 @@ wheels = [
 
 [[package]]
 name = "floe-python"
-version = "0.8.1"
+version = "0.8.2"
 source = { editable = "." }
 dependencies = [
     { name = "boto3" },