floe-python 0.8.0__tar.gz → 0.8.2__tar.gz

{floe_python-0.8.0 → floe_python-0.8.2}/CHANGELOG.md

@@ -2,6 +2,104 @@
 
 All notable changes to Floe are documented in this file.
 
+## Unreleased
+
+### dagster-floe
+
+- **`ManifestEntity` exposes `source_resource` (#1044)**, the entity's `source.resource`, next to
+  `source_ref`, so a `source_key_fn` can build `<source>/<resource>` from the manifest.
+
+- **Accepted and rejected assets carry `floe/entity` and `floe/output` metadata (#1045)**: the entity
+  name and `"accepted"` or `"rejected"`, so callers no longer parse the `<op>__accepted` output names.
+
+- **Breaking: rejected assets are keyed `<domain>/rejected/<table>` (#1046).** They were
+  `<domain>/<table>_rejected`, which collided with a real table named `<table>_rejected`. Upgrade:
+  the history of the rejected assets restarts under the new key, and any selection, sensor or
+  automation condition that names the old keys must be updated.
+
+- **The `floe/table` asset tag of a Glue entity expands `{domain}` in `database` (#1043)**, so it
+  names the database the run writes.
+
+### airflow-floe
+
+- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
+  dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
+
+## v0.8.2
+
+- **A Glue catalog can write one database per domain (#1043).** `database` may contain `{domain}`,
+  e.g. `dev_{domain}_silver`: per entity it expands to `sink.accepted.iceberg.namespace`, else
+  `entity.domain`, so one catalog writes `dev_sales_silver` and `dev_finance_silver`. Put the catalog in
+  each stage's profile (`catalogs:` replaces the config's) to vary the prefix per stage. An entity
+  with neither is a `floe validate` error. A `database` without `{domain}` behaves as before.
+
+## v0.8.1
+
+- **floe-python requires `boto3>=1.36.0` (was `>=1.43.118`) (#1028).** The old floor made `s3fs`
+  uninstallable next to Floe: current `aiobotocore` caps `botocore` below 1.43.118. 1.36.0 is the
+  first release with every S3 call and option Floe uses (conditional writes, checksum settings).
+
+- **AWS Glue catalogs are back.** Floe 0.8.0 rejected `catalogs.definitions[].type: glue`, a
+  regression for AWS deployments (OpenLakeForge). A Glue catalog takes the 0.7 keys (`region`,
+  `database`, `create_database_if_missing`, `warehouse_storage`, `warehouse_prefix`) and writes
+  Iceberg tables through PyIceberg's Glue catalog, with credentials from the AWS chain.
+  `overwrite` and `append` are supported; `merge_scd1`/`merge_scd2` and `floe maintain --compact`
+  still need a REST catalog. **D42:** `allow_takeover` is accepted and ignored; 0.8 does not check
+  0.7's `floe.managed` ownership marker.
+
+- **`floe run --domain <d>` (#1024)** selects every entity of a domain, for `-c` and `--manifest` runs. It
+  takes several domains (repeatable or comma-separated), fails on an unknown domain, and cannot be combined
+  with `--entities`.
+
+- **`kubernetes_job` runners can shape the Job's pod (#1023).** `execution.runner` in a profile accepts
+  `labels` (added to the defaults), `annotations`, `image_pull_policy`, `security_context`,
+  `node_selector`, `tolerations`, `backoff_limit` and `env_from`. They are carried into the manifest
+  runner definition only when set, so existing manifests are unchanged. dagster-floe 0.3.1 applies them;
+  airflow-floe ignores them.
+
+- **A profile's `execution.runner.command` may be a list (#1034)**, e.g. `command: [olf-floe-job, --]`,
+  as the manifest and the connectors already accept. A string still works.
+
+- **`-p s3://...`.** `-p` / `--profile` and `profile_path=` accept an `s3://` URI, read as an `s3://` manifest is,
+  so a container run no longer needs the profile file copied in (#1025). `floe manifest generate -p s3://...`
+  records the URI as `profile_uri`. `-c` still takes a local path.
+
+## dagster-floe v0.3.1
+
+- **The `floe` extra requires `floe-python>=0.8.1,<0.9` (#1041)**, the first release with `floe run --domain`.
+
+- **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031).** It
+  used to replace the whole argument list, so every Job ran every entity. `args` now prefix Floe's
+  rendered arguments: `command: [python]` + `args: [-m, floe]` runs
+  `python -m floe run … --entities …`.
+
+- **`s3://` run reports and manifests are read with boto3 (#1021).** When boto3 is importable (it is whenever the
+  `floe` extra is installed) it replaces fsspec/s3fs for `s3://`, honouring `AWS_ENDPOINT_URL_S3` /
+  `AWS_ENDPOINT_URL`, `AWS_REGION` and `AWS_S3_FORCE_PATH_STYLE`. s3fs no longer has to be installable next to
+  `floe-python`; without it Silver assets lost row counts, rejection counts and check detail. Other schemes, and
+  `s3://` without boto3, still use fsspec.
+
+- **`kubernetes_job` pod options are applied to the Job (#1023).** dagster-floe puts the runner's `labels`
+  (added to `app` and `floe-job`, which are never overridden), `annotations`, `image_pull_policy`,
+  `security_context`, `node_selector`, `tolerations`, `backoff_limit` (default 0) and `env_from` in the Job
+  it creates. With retries the run waits for the Job's final status and reads the pod that matches it.
+
+- **`manifest_name=` in config mode (#1022).** `load_floe_assets`, `build_definitions` and
+  `build_job_run_config_from_manifest` accept `manifest_name=` next to `config_path=` and pass it to
+  `floe.build_manifest`. With a fixed name, the op name of a `job_scope: domain` asset stays
+  `floe_<name>__<domain>__...` when the config changes; without it the name derives from the config's path and
+  checksum, as before. Passing it with `manifest_path=` raises `ValueError`.
+
+- **`source_key_fn` picks each entity's upstream source key (#1020).** `load_floe_assets` and
+  `build_definitions` accept `source_key_fn=lambda entity: AssetKey([...])` (or `None` for the default key).
+  Entities of one named source can depend on different upstream assets, e.g. `crm/orders` and `crm/accounts`,
+  and Dagster still orders each after its upstream. No `SourceAsset` is registered for a key you choose.
+
+- **`job_scope: domain` Jobs run `floe run --domain <d>` for a whole domain (#1036).** When the selected
+  entities are exactly every entity of the domain in the manifest, the Job sends `--domain <d>` in place of
+  `--entities a,b,c`; a subset keeps `--entities`. It needs Floe 0.8.1 (`floe run --domain`):
+  a `kubernetes_job` image must be 0.8.1 or newer.
+
 ## v0.8.0
 
 Floe 0.8 is the Python core. `pip install floe-python` installs the `floe` command (a pure-Python

{floe_python-0.8.0 → floe_python-0.8.2}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.0
+Version: 0.8.2
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11
-Requires-Dist: boto3>=1.43.118
+Requires-Dist: boto3>=1.36.0
 Requires-Dist: duckdb==1.5.6
 Requires-Dist: fastexcel>=0.21.0
 Requires-Dist: polars>=2.0.0

{floe_python-0.8.0 → floe_python-0.8.2}/README.md

@@ -45,10 +45,10 @@ table history through DuckDB, all on a single node.
 - **Inputs:** CSV · TSV · JSON · Parquet · ORC · Avro · XLSX · XML
 - **Accepted outputs:** Parquet · Apache Iceberg
 - **Storage:** local · S3 (and S3-compatible stores)
-- **Catalogs:** Iceberg REST (Polaris)
+- **Catalogs:** Iceberg REST (Polaris) · AWS Glue
 
-Floe 0.8 dropped the Delta, DuckDB and fixed-width formats, the Glue and Unity
-catalogs, and ADLS and GCS storage. The Rust implementation that supported them
+Floe 0.8 dropped the Delta, DuckDB and fixed-width formats, the Unity
+catalog, and ADLS and GCS storage. The Rust implementation that supported them
 (0.7 and earlier) is archived at
 [OpenLakeForge/floe-rust](https://github.com/OpenLakeForge/floe-rust).
 

{floe_python-0.8.0 → floe_python-0.8.2}/conformance/ARCHIVE.md

@@ -187,9 +187,9 @@ implementation exists yet; each is derived as its row says.
 | `d1_sink_delta` | D1 | `entity.name=orders sink.accepted.format=delta is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
 | `d1_sink_duckdb` | D1 | `entity.name=orders sink.accepted.format=duckdb is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
 | `d1_catalog_unity` | D1 | `catalogs.definitions name=u type=unity is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
-| `d1_catalog_glue` | D1 | `catalogs.definitions name=g type=glue is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
-| `parse_catalogs_type_unsupported` | D1 | `catalogs.definitions[0]: catalogs.definitions name=c has unsupported type=hive (supported: rest)` | Hand-derived, not recorded: Rust's message with the supported list reduced to `rest`, since Glue and Unity are dropped features (D1). |
-| `prof_catalogs_definition_type_unsupported` | D1 | `profile.catalogs.definitions[0]: profile.catalogs.definitions name=r has unsupported type=hive (supported: rest)` | Hand-derived, not recorded: Rust's message with the supported list reduced to `rest`, since Glue and Unity are dropped features (D1). |
+| `d1_catalog_glue` | D1 | `ok` | Glue was dropped in 0.8.0 (D1) and restored in 0.8.1: the 0.7 config validates again, as in Rust. |
+| `parse_catalogs_type_unsupported` | D1 | `catalogs.definitions[0]: catalogs.definitions name=c has unsupported type=hive (supported: glue, rest)` | Hand-derived, not recorded: Rust's message with `unity` dropped from the supported list, since Unity is a dropped feature (D1). |
+| `prof_catalogs_definition_type_unsupported` | D1 | `profile.catalogs.definitions[0]: profile.catalogs.definitions name=r has unsupported type=hive (supported: glue, rest)` | Hand-derived, not recorded: Rust's message with `unity` dropped from the supported list, since Unity is a dropped feature (D1). |
 | `d1_storage_adls` | D1 | `storages.definitions name=a type=adls is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
 | `d1_storage_gcs` | D1 | `storages.definitions name=g type=gcs is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |
 | `d1_source_fixed` | D1 | `entity.name=orders source.format=fixed is not supported in 0.8` | D1 (dropped feature, OLF scope): Python rejects it at validate. Not recorded from Rust. |

{floe_python-0.8.0 → floe_python-0.8.2}/conformance/config/parse_catalogs_type_unsupported/expected/validate.json

@@ -1,4 +1,4 @@
 {
   "ok": false,
-  "error": "catalogs.definitions[0]: catalogs.definitions name=c has unsupported type=hive (supported: rest)"
+  "error": "catalogs.definitions[0]: catalogs.definitions name=c has unsupported type=hive (supported: glue, rest)"
 }

{floe_python-0.8.0 → floe_python-0.8.2}/conformance/config/prof_catalogs_definition_type_unsupported/expected/validate.json

@@ -1,4 +1,4 @@
 {
   "ok": false,
-  "error": "profile.catalogs.definitions[0]: profile.catalogs.definitions name=r has unsupported type=hive (supported: rest)"
+  "error": "profile.catalogs.definitions[0]: profile.catalogs.definitions name=r has unsupported type=hive (supported: glue, rest)"
 }

floe_python-0.8.2/conformance/config/val2_uk_ok/expected/validate.json

@@ -0,0 +1,3 @@
+{
+  "ok": true
+}

{floe_python-0.8.0 → floe_python-0.8.2}/context/orchestrators.md

@@ -204,13 +204,21 @@ When `type = kubernetes_job`, the manifest runner definition supports:
 | `domain_resources` | object | Resources per domain name; used by `domain` Jobs in place of `resources` (no key-wise merge) |
 | `env` | object | Plain env-var key→value pairs injected into the container |
 | `secrets` | array | Kubernetes Secret key references: `[{name, secret_name, key}]` |
+| `labels` | object | Pod labels added to the defaults (`app`, `floe-job`); the defaults win on a clash, because `floe-job` is how the pod is found again |
+| `annotations` | object | Pod template annotations |
+| `image_pull_policy` | string | Container `imagePullPolicy` |
+| `security_context` | object | Container `securityContext`, verbatim |
+| `node_selector` | object | Pod `nodeSelector` |
+| `tolerations` | array | Pod `tolerations`, verbatim |
+| `backoff_limit` | integer >= 0 | Job `backoffLimit` (default `0`) |
+| `env_from` | array | Container `envFrom`, verbatim, e.g. `[{secretRef: {name: floe-env}}]` |
 
 These fields are populated automatically when `floe manifest generate` is given a profile with a `kubernetes_job` runner block. The `dagster-floe[kubernetes]` optional extra expresses the Kubernetes client requirement.
 
 ### Job scope (`job_scope`)
 
 - **`entity`** (default, also when unset): one Job per entity, named `floe-<entity>-<id>`. Pod start is paid once per entity; each entity has its own timeout, TTL and failure. Use it for a few large entities, or when entities need isolation.
-- **`domain`**: entities of one domain whose runner has `job_scope: domain` form one subsettable multi-asset per (domain, runner), so an entity with another runner gets its own Job. Materializing it launches **one** Job named `floe-<domain>-<id>` running `floe run --manifest <uri> --entities a,b,c` for the selected entities in manifest order, one after another, so Polaris never sees overlapping commits from the Job. The Job is sized from `domain_resources[<domain>]`, else `resources`; `timeout_seconds`, TTL and polling apply to the one Job. Use it when pod start dominates (many small entities). Select the accepted assets: a rejected asset is materialized only when it is selected too, and the asset job selects both. The `floe/table` op tag (#924) is not set on a domain asset.
+- **`domain`**: entities of one domain whose runner has `job_scope: domain` form one subsettable multi-asset per (domain, runner), so an entity with another runner gets its own Job. Materializing it launches **one** Job named `floe-<domain>-<id>` running `floe run --manifest <uri> --entities a,b,c` (`--domain <d>` when they are every entity of the domain in the manifest) for the selected entities in manifest order, one after another, so Polaris never sees overlapping commits from the Job. The Job is sized from `domain_resources[<domain>]`, else `resources`; `timeout_seconds`, TTL and polling apply to the one Job. Use it when pod start dominates (many small entities). Select the accepted assets: a rejected asset is materialized only when it is selected too, and the asset job selects both. The `floe/table` op tag (#924) is not set on a domain asset.
 
 Dagster still reports per entity (materialization, rows, rejected counts, report URI, asset checks) from the run's `run_finished` event and summary.
 

{floe_python-0.8.0 → floe_python-0.8.2}/context/project.md

@@ -43,9 +43,9 @@ Local · AWS S3 (and S3-compatible stores)
 
 ## Supported catalogs
 
-Iceberg REST (Polaris)
+Iceberg REST (Polaris) and AWS Glue (restored in 0.8.1; overwrite and append only)
 
-Floe 0.8 dropped Delta, DuckDB/MotherDuck, fixed-width, Glue, Unity, ADLS and GCS (D1, D26 in
+Floe 0.8 dropped Delta, DuckDB/MotherDuck, fixed-width, Unity, ADLS and GCS (D1, D26 in
 #619). The Rust implementation that had them is archived at
 [OpenLakeForge/floe-rust](https://github.com/OpenLakeForge/floe-rust).
 

{floe_python-0.8.0 → floe_python-0.8.2}/docs/cli.md

@@ -9,7 +9,7 @@
   - Validate YAML config and exit with non-zero code on errors.
   - `-p` applies an environment profile first. With `-p` and no `-c`, only the profile is validated.
 
-- `floe run (-c <config> | --manifest <manifest.json>) [-p <profile>] [--run-id <id>] [--entities <name[,name...]>] [-F|--full-refresh] [--dry-run] [--log-format off|text|json] [--quiet|--verbose]`
+- `floe run (-c <config> | --manifest <manifest.json>) [-p <profile>] [--run-id <id>] [--entities <name[,name...]>] [--domain <name[,name...]>] [-F|--full-refresh] [--dry-run] [--log-format off|text|json] [--quiet|--verbose]`
   - Execute ingestion using the config, or a manifest generated by `floe manifest generate`.
   - Run reports are written under `<report.path>/run_<run_id>/...` as defined in the config,
     or `<report.path>/<domain>/run_<run_id>/...` when every selected entity has one domain (D37).
@@ -19,6 +19,9 @@
     `image` manifest or the current directory for a `cli` one. An `s3://` `report_base_uri` writes the
     reports there.
   - `-p` applies an environment profile (ignored with `--manifest`: the manifest is already compiled).
+    It takes a local path or an `s3://` URI, read as `--manifest` is (every command that takes `-p` does the same).
+  - `--domain` selects every entity of the named domains (repeatable or comma-separated), in config order.
+    An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state
     already lists are read again, `append` writes as `overwrite`, and with `sink.archive` the archived files are read
     too (D24). It is rejected with merge modes.
@@ -89,6 +92,8 @@
 
 - Run with default paths from the config:
   - `floe run -c example/config.yml --entities customer`
+- Run every entity of a domain:
+  - `floe run -c example/config.yml --domain sales`
 - Full refresh of one entity:
   - `floe run -c example/config.yml --entities customer --full-refresh`
 - Run from a manifest:

{floe_python-0.8.0 → floe_python-0.8.2}/docs/config.md

@@ -91,8 +91,8 @@ entities:
 - `catalogs` (optional)
   - Defines named Iceberg REST catalogs (Polaris, Nessie, Snowflake Open Catalog, etc.).
     An Iceberg accepted sink requires one (D26).
-  - Supported type: `rest`. `glue` and `unity` were dropped in 0.8 (D1) and fail validation;
-    the archived [OpenLakeForge/floe-rust](https://github.com/OpenLakeForge/floe-rust) documents them for 0.7.
+  - Supported types: `rest` and `glue` (AWS Glue, restored in 0.8.1). `unity` was dropped in 0.8
+    (D1) and fails validation; the archived [OpenLakeForge/floe-rust](https://github.com/OpenLakeForge/floe-rust) documents them for 0.7.
   - Keys:
     - `default` (optional): default catalog name used by `sink.accepted.iceberg.catalog` when omitted.
     - `definitions` (required): array of catalog definitions.
@@ -105,6 +105,11 @@ entities:
       - `scope` (optional): OAuth2 scope
       - `warehouse_storage` (optional): storage name for table location resolution; must be an S3 storage
       - `warehouse_prefix` (optional): prefix for deterministic table locations
+    - Glue definitions (`type: glue`) take `name`, `region` (required), `database` (required:
+      the Glue database that holds the tables; `{domain}` expands per entity to its
+      `sink.accepted.iceberg.namespace` or `domain`), `create_database_if_missing` (optional, default
+      `true`), `warehouse_storage` (must be an S3 storage) and `warehouse_prefix`. Glue supports
+      `overwrite` and `append`; merges need a REST catalog. See [Iceberg sink](sinks/iceberg.md).
 - `lineage` (optional)
   - Enables OpenLineage event emission for this run.
   - `url` (required): base URL of the OpenLineage-compatible endpoint (e.g. `http://marquez:5000`)

{floe_python-0.8.0 → floe_python-0.8.2}/docs/manifest.md

@@ -189,7 +189,8 @@ A manifest is a self-contained JSON document. Here's an annotated excerpt:
     }
   },
 
-  // Runner definitions (local, kubernetes, databricks, etc.)
+  // Runner definitions (local, kubernetes, databricks, etc.). A `kubernetes_job` definition also
+  // carries the profile's pod options (`labels`, `security_context`, ...) when set; see profiles.md.
   "runners": {
     "default": "local",
     "definitions": {

{floe_python-0.8.0 → floe_python-0.8.2}/docs/profiles.md

@@ -49,6 +49,22 @@ All other fields are optional.
 
 ---
 
+## Profile location
+
+A profile is a local path or an `s3://` URI, for `-p` / `--profile` on every command and for `profile_path=`
+in the Python API:
+
+```bash
+floe run -c config.yml -p s3://my-bucket/profiles/prod.yml
+```
+
+An `s3://` profile is read with the AWS default credential chain (`AWS_ENDPOINT_URL_S3` and
+`AWS_S3_FORCE_PATH_STYLE` for S3-compatible stores), as an `s3://` manifest is. A missing object fails with
+`profile not found: <uri>`. `floe manifest generate -p s3://...` records the URI as `profile_uri`, and the
+SHA-256 of the object as `profile_checksum`. The config (`-c`) is still a local path.
+
+---
+
 ## Variable precedence and resolution
 
 When variable sources are merged at runtime the following precedence applies
@@ -190,6 +206,31 @@ Per-run env vars and `existing_cluster_id`:
   process still reads `env_parameters.FLOE_ENV` to render `{env}` in
   `job_name`, but does not push it to the running task.
 
+### kubernetes.yml (pod options)
+
+A `kubernetes_job` runner can shape the Job's pod. Every option is optional and is carried into the
+manifest runner definition only when set; dagster-floe applies them (airflow-floe ignores them).
+
+```yaml
+execution:
+  runner:
+    type: kubernetes_job
+    image: registry/floe:0.8
+    namespace: floe-prod
+    command: [python, -m, floe]              # container command, a string or a list
+    labels: {olf.io/flow: ingest}            # added to the defaults (`app`, `floe-job`), which win on a clash
+    annotations: {team: data}                # pod template annotations
+    image_pull_policy: IfNotPresent
+    security_context: {runAsNonRoot: true}   # container securityContext, verbatim
+    node_selector: {pool: batch}
+    tolerations: [{key: batch, operator: Exists, effect: NoSchedule}]   # verbatim
+    backoff_limit: 2                         # Job backoffLimit, default 0
+    env_from: [{secretRef: {name: floe-env}}]   # container envFrom, verbatim
+```
+
+`labels`, `annotations` and `node_selector` are string maps, `backoff_limit` is a non-negative integer,
+and `security_context`, `tolerations` and `env_from` are copied as written: Kubernetes validates them.
+
 ---
 
 ## Using profile variables in a Floe config
@@ -262,7 +303,8 @@ Profiles can carry a `catalogs:` block with the same catalog definition shape as
 main Floe configs. When a command is run with both `--config` and `--profile`,
 the profile `catalogs:` block replaces the config-level `catalogs:` block for
 that invocation. This keeps entity logic in the shared config while moving
-REST Iceberg catalog endpoints (the only catalog type in 0.8) into environment-specific files.
+Iceberg catalog endpoints and Glue databases (e.g. `database: dev_{domain}_silver`) into
+environment-specific files.
 
 ```yaml
 apiVersion: floe/v1

{floe_python-0.8.0 → floe_python-0.8.2}/docs/python-bindings.md

@@ -146,7 +146,7 @@ All events have an `event` key (string discriminator) and `ts_ms` (Unix timestam
 Profiles let you override config variables, storage credentials, or lineage settings without editing the YAML. There are two ways to provide them:
 
 ```python
-# Option 1: load a profile YAML file
+# Option 1: load a profile YAML file (a local path or an s3:// URI)
 floe.run("orders.yml", profile_path="prod.yml")
 
 # Option 2: pass overrides inline (useful in notebooks and CI)

{floe_python-0.8.0 → floe_python-0.8.2}/docs/sinks/iceberg.md

@@ -67,6 +67,35 @@ Floe passes the S3 storage's `endpoint`, `region` and `path_style_access` to PyI
 DuckDB. Credentials come from the standard AWS credential chain (environment variables, web
 identity / IRSA, container credentials, instance metadata); the catalog does not vend them.
 
+## AWS Glue catalog
+
+A `type: glue` catalog writes the table through the AWS Glue Data Catalog, as Floe 0.7 did:
+
+```yaml
+catalogs:
+  default: "glue"
+  definitions:
+    - name: "glue"
+      type: "glue"
+      region: "eu-west-1"
+      database: "sales_silver"           # the Glue database that holds the tables
+      warehouse_storage: "lakehouse_silver"
+      warehouse_prefix: "warehouse/iceberg"
+      create_database_if_missing: false  # default true
+```
+
+The table is `<database>.<table>`; `sink.accepted.iceberg.namespace` (or `entity.domain`) only
+names its location, `<warehouse_prefix>/<namespace>/<table>`. To write one database per domain from
+a single catalog, put `{domain}` in `database`, e.g. `database: dev_{domain}_silver`: per
+entity it becomes `sink.accepted.iceberg.namespace`, else `entity.domain`, normalized like the
+namespace (lowercase, `[a-z0-9_-]`), so the `sales` and `finance` domains write to `dev_sales_silver`
+and `dev_finance_silver`. An entity with neither a namespace nor a domain is a `floe validate`
+error. `{{VAR}}` placeholders are not applied to catalog definitions: to vary the prefix per stage,
+define the catalog in each stage's profile ([profile catalogs](../profiles.md#environment-catalog-overrides)
+replace the config's). Glue credentials come from the AWS credential chain, as for S3. Glue supports `overwrite` and `append`: `floe validate` rejects a
+merge write mode, and `floe maintain --compact` needs a REST catalog. `allow_takeover` is
+accepted and ignored: Floe 0.8 writes any Iceberg table in the database (D42).
+
 ## REST catalog credentials
 
 REST catalog definitions accept bearer tokens and OAuth2 client credentials:
@@ -119,7 +148,7 @@ Current scope:
 
 ## Supported scope (current)
 
-- Accepted sink only (`sink.accepted.format: iceberg`), through a REST catalog
+- Accepted sink only (`sink.accepted.format: iceberg`), through a REST or AWS Glue catalog
 - S3 storage
 - `sink.write_mode`: `overwrite`, `append`, `merge_scd1` and `merge_scd2`. See [write modes](../write_modes.md).
 - Each entity's write is one catalog commit. `overwrite` replaces every existing data file in

{floe_python-0.8.0 → floe_python-0.8.2}/docs/summary.md

@@ -65,8 +65,8 @@ CLI reference, including `manifest generate`, `add-entity`, `state inspect`,
 
 - Floe runs as a lightweight single-node Python runtime using Polars, PyIceberg
   and DuckDB.
-- Floe 0.8 dropped the Delta, DuckDB and fixed-width formats, the Glue and Unity
-  catalogs, and ADLS and GCS storage (see [support-matrix.md](support-matrix.md)).
+- Floe 0.8 dropped the Delta, DuckDB and fixed-width formats, the Unity
+  catalog, and ADLS and GCS storage (see [support-matrix.md](support-matrix.md)).
 - File-based incremental ingestion is implemented. Row-based incremental mode
   remains a contract-level value for future work.
 - Floe reports write-time metrics and metadata for accepted outputs where the

{floe_python-0.8.0 → floe_python-0.8.2}/docs/support-matrix.md

@@ -14,7 +14,6 @@ These 0.7 features are rejected by `floe validate` with a "not supported in 0.8"
 | Fixed-width input (`source.format: fixed`) | Convert to CSV upstream |
 | Delta Lake accepted sink and Unity Catalog | Iceberg |
 | DuckDB / MotherDuck accepted sink | Iceberg (merges run through DuckDB) |
-| AWS Glue catalog | Iceberg REST catalog (Polaris) |
 | Catalog-less Iceberg tables (filesystem catalog) | Iceberg REST catalog |
 | ADLS and GCS storage | S3 or an S3-compatible store |
 
@@ -49,15 +48,15 @@ Notes:
 | Output | Local | S3 | Notes |
 |---|---|---|---|
 | Accepted: Parquet | ✅ | ✅ (upload) | Writes `part-*.parquet`; `overwrite` and `append` |
-| Accepted: Iceberg | ❌ | ✅ (REST catalog) | `overwrite`, `append`, `merge_scd1` and `merge_scd2` in one catalog commit per entity |
+| Accepted: Iceberg | ❌ | ✅ (REST or Glue catalog) | `overwrite`, `append`, `merge_scd1` and `merge_scd2` in one catalog commit per entity; merges need a REST catalog |
 | Rejected: CSV | ✅ | ✅ (upload) | Dataset parts `part-*.csv`; the only rejected format |
 | Reports: JSON | ✅ | ✅ (upload) | |
 
 Notes:
 - `sink.write_mode` applies to accepted and rejected outputs (`overwrite`, `append`,
   `merge_scd1`, `merge_scd2`). Merge modes need an Iceberg accepted sink.
-- An Iceberg accepted sink requires a REST catalog in `catalogs.definitions` (D26) and S3
-  storage for the table data.
+- An Iceberg accepted sink requires a REST or AWS Glue catalog in `catalogs.definitions` (D26)
+  and S3 storage for the table data.
 - Iceberg `partition_spec` supports `identity`, `year`, `month`, `day`, `hour` and
   `bucket` (with `num_buckets`, D12).
 - `schema.schema_evolution.mode: add_columns` is Iceberg-only and additive-only (D13).

{floe_python-0.8.0 → floe_python-0.8.2}/orchestrators/airflow-floe/src/airflow_floe/k8s_status.py

@@ -22,6 +22,10 @@ def map_k8s_job_status(job: Any) -> str:
     succeeded = _read(status, "succeeded", 0)
     failed = _read(status, "failed", 0)
     conditions = _read(status, "conditions", []) or []
+    spec = job.get("spec") if isinstance(job, dict) else getattr(job, "spec", None)
+    backoff_limit = _read(spec, "backoff_limit", _read(spec, "backoffLimit", 0))
+    if not isinstance(backoff_limit, int):
+        backoff_limit = 0
 
     for condition in conditions:
         cond_type = _read(condition, "type", "")
@@ -37,7 +41,7 @@ def map_k8s_job_status(job: Any) -> str:
 
     if succeeded > 0:
         return STATUS_SUCCEEDED
-    if failed > 0:
+    if failed > backoff_limit:  # a failed pod is retried until the Job's `Failed` condition
         return STATUS_FAILED
     if active > 0:
         return STATUS_RUNNING

{floe_python-0.8.0 → floe_python-0.8.2}/orchestrators/airflow-floe/src/airflow_floe/kubernetes_runner.py

@@ -62,8 +62,8 @@ def build_k8s_job_spec(
 
     if runner.command is not None:
         container_cmd = runner.command
-    if runner.args is not None:
-        container_args = runner.args
+    if runner.args is not None:  # a prefix: replacing the list would drop the entity selection
+        container_args = runner.args + container_args
 
     # Env vars: regular key/value first, then secret refs.
     env_vars: list[dict[str, Any]] = []

{floe_python-0.8.0 → floe_python-0.8.2}/orchestrators/airflow-floe/tests/test_kubernetes_runner.py

@@ -217,13 +217,13 @@ class BuildJobSpecTests(unittest.TestCase):
 
     def test_full_spec_with_all_optional_fields(self) -> None:
         runner = _full_runner()
-        spec = build_k8s_job_spec("floe-full-abc", ["ignored"], runner)
+        spec = build_k8s_job_spec("floe-full-abc", ["floe", "run", "--entities", "orders"], runner)
 
         container = spec["spec"]["template"]["spec"]["containers"][0]
 
-        # Command/args overrides from runner definition
+        # Command override replaces the binary; args prefix the rendered arguments
         self.assertEqual(container["command"], ["python"])
-        self.assertEqual(container["args"], ["-m", "floe"])
+        self.assertEqual(container["args"], ["-m", "floe", "run", "--entities", "orders"])
 
         # Service account
         self.assertEqual(

{floe_python-0.8.0 → floe_python-0.8.2}/orchestrators/dagster-floe/README.md

@@ -19,7 +19,8 @@ Current model:
 - Asset generation from `entities[]` (`asset_key`, `group_name` respected).
 - One upstream source asset per entity (`<entity>_source`), or one shared `sources/<project>/<name>` asset per
   named source of the manifest `sources` index (group `sources`; `<project>` is `manifest_name`, else `manifest_id`).
-- Manifests load from a local path or a bucket URI (`s3://...`, `remote` extra; AWS default credential chain).
+- Manifests load from a local path or a bucket URI (`s3://...` with the `floe` extra's boto3, other schemes with the
+  `remote` extra; AWS default credential chain).
 - One Dagster job per manifest (stable job names).
 - Multi-manifest loading (`*.manifest.json`) with collision checks.
 - Local runner (`local_process`) support.
@@ -29,19 +30,20 @@ Current model:
 ## Install
 
 Prereqs:
-- Floe 0.8 (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
+- Floe 0.8.1 or newer (`pip install floe-python`, **or** Docker with the `ghcr.io/openlakeforge/floe` image).
 - Python 3.10+.
 
 ```bash
 pip install dagster-floe
-pip install "dagster-floe[floe]"   # also installs floe-python>=0.8,<0.9 for the in-process runner
+pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.1,<0.9 for the in-process runner
 ```
 
 ## Supported Floe versions
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
-| 0.3.x | 0.8 | `0.8` |
+| 0.3.1 | 0.8.1 or newer (`job_scope: domain` Jobs run `floe run --domain`) | `0.8` |
+| 0.3.0 | 0.8 | `0.8` |
 | 0.2.x | 0.7 and earlier | see 0.2.x docs |
 
 ## Development install (from this repo)
@@ -85,6 +87,32 @@ defs = build_definitions(config_path="/app/floe.yml", profile_path="/app/prod.ym
   place of `--manifest <uri>`, both in process and as a local subprocess or `kubernetes_job` pod. The config
   and profile paths must therefore exist where the run happens; for a `kubernetes_job`, that's the run image.
 
+In config mode, pass `manifest_name="sales"` (to `load_floe_assets`, `build_definitions` or
+`build_job_run_config_from_manifest`) to keep the op name of a `job_scope: domain` asset stable. Without it the name
+derives from the config's path and checksum, so every edit of the config renames the op and breaks its history in
+Dagster. The job name (`floe_<manifest_id>_job`) still derives from the manifest id; pin it with
+`build_definitions(job_name=...)`.
+
+## Choose each entity's upstream key
+
+Entities of a source named in the manifest `sources` index share one `sources/<project>/<source>` dependency.
+Pass `source_key_fn` to `load_floe_assets` or `build_definitions` to give each entity its own upstream asset
+instead (for example, to depend on `crm/orders` and `crm/accounts` in domain mode). `entity.source_ref` and
+`entity.source_resource` carry the entity's `source.ref` and `source.resource` (`None` when absent):
+
+```python
+from dagster import AssetKey
+
+defs = build_definitions(
+    manifest_path="manifests/sales.json",
+    source_key_fn=lambda e: AssetKey([e.source_ref, e.source_resource]) if e.source_resource else None,  # None: default key
+)
+```
+
+The function receives each manifest entity and returns an `AssetKey`, or `None` for the default. The key is a
+dependency of the entity's asset like the default one, so Dagster still orders the asset after its upstream.
+You own the asset behind a key you choose: Floe registers no `SourceAsset` for it, so define it in your code.
+
 ## Run the example (repo-only)
 
 ```bash
@@ -113,8 +141,9 @@ Checks appear in the Dagster UI under each asset, linked to the run that produce
 
 ## Reading manifests and reports from S3
 
-`dagster-floe` reads `s3://` manifests and Floe run reports with `fsspec` (install the `remote`
-extra). It configures S3 the same way the Floe run does, from the environment:
+`dagster-floe` reads `s3://` manifests and Floe run reports with boto3, which the `floe` extra installs
+(without boto3 it falls back to `fsspec` and the `remote` extra). It configures S3 the same way the Floe
+run does, from the environment:
 
 | Variable | Effect |
 | --- | --- |
@@ -152,6 +181,10 @@ floe manifest generate -c orders.yml -p prod.yml --output manifests/orders.json
 
 See [docs/lineage.md](../../docs/lineage.md) for the full lineage config reference.
 
+## Output metadata
+
+Every accepted and rejected asset carries `floe/entity` (the manifest entity name) and `floe/output` (`"accepted"` or `"rejected"`) in its asset metadata, so tooling can tell the two apart without parsing dagster-floe's output names: `spec.metadata["floe/output"]` on each `AssetSpec` of `multi_asset.specs`.
+
 ## Notes
 
 - This connector does **not** parse YAML directly; it consumes Floe manifests (`spec_version` `0.8`), read from a file or built by `floe.build_manifest` in config mode.

{floe_python-0.8.0 → floe_python-0.8.2}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.3.0"
+version = "0.3.1"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"
@@ -33,7 +33,7 @@ kubernetes = [
   "kubernetes>=28",
 ]
 floe = [
-  "floe-python>=0.8,<0.9",
+  "floe-python>=0.8.1,<0.9",
 ]
 remote = [
   "fsspec",