floe-python 0.8.7__tar.gz → 0.8.9__tar.gz

{floe_python-0.8.7 → floe_python-0.8.9}/CHANGELOG.md

@@ -2,6 +2,57 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.9
+
+- **A file removed after listing no longer aborts a run with `sink.rejected` set (#1121).** The file is reported as
+  failed with its read error, without a raw copy, and the run continues.
+
+- **Tied unique-key samples are ordered by the key, not its Rust Debug text (#984).** When duplicate keys share a
+  count, the report samples now order them by value: numbers numerically (2 before 10, no longer 10 before 2),
+  strings by code point, and composite keys column by column. Reports whose tied samples held keys of different
+  digit counts, or strings with control characters, list them in a different order.
+
+- **Iceberg sinks apply `options.compression` and `options.row_group_size` (#1009).** Both were parsed and
+  validated but ignored for `sink.accepted.format: iceberg`. `overwrite` and `append` now write data files with the
+  configured codec and rows per row group, for that write only; the table's properties are not changed. Merge modes
+  still ignore them.
+
+- **Parallel runs list entity inputs on the worker pool (#1131).** With `workers` > 1, the inputs of all entities are
+  listed on up to `workers` threads instead of one after another; they are still all resolved before the first
+  `entity_started`, and a listing failure raises the first error in config order.
+
+- **Opening an existing Iceberg table takes one catalog call instead of three (#1130).** `ensure_table` now tries
+  `load_table` first and checks or creates the namespace only when the table is missing; merge entities no longer
+  call `table_exists` and `load_table` before it. With a remote catalog this saves about 0.2 s per entity at
+  100 ms of latency. Errors and messages are unchanged.
+
+- **Overwriting the same Iceberg table no longer gets slower with every earlier overwrite (#1132).** PyIceberg 0.12
+  left the all-deleted manifest of each earlier overwrite in the table's manifest list, so every scan and commit read
+  one more manifest. On a local SQL catalog the 100th overwrite took about 430-750 ms against about 45 ms for the
+  2nd; it now takes about 230 ms, and the manifest list stays at 2 entries. A table that already accumulated them
+  sheds them on its next overwrite.
+
+## v0.8.8
+
+- **Entities can run in parallel in one `floe run` (#1105).** With `execution.orchestration.strategy: parallel` in
+  the profile, `floe run -c … -p …`, `floe run --manifest …` and `floe.run(…, profile_path=…)` run up to
+  `max_concurrent_entities` entities at once (default `min(4, CPU count)`). Entities that share an Iceberg table, an
+  accepted path or a state file never run together, and merge entities run one at a time. Reports and the summary
+  stay in config order; events of different entities interleave and observers are called from worker threads. On
+  the Polaris/SeaweedFS harness, 12 entities writing 12 Iceberg tables took 2.04 s with one worker and 1.03 s with
+  four. Without `strategy: parallel` nothing changes. Profiles that already set `strategy: parallel` for Dagster now
+  also run the entities of a `job_scope: domain` Job in parallel; peak memory then grows with the number of workers
+  (see `docs/profiles.md`).
+
+- **Creating a new Iceberg namespace no longer fails when two writers race (#1117).** Two entities, or two
+  processes such as concurrent Dagster Jobs, that both found a namespace missing and created it made the second
+  creation fail the run. A namespace that already exists by then now counts as created.
+
+- **A small JSON file is parsed once (#1115).** The key check before the read parsed every JSON file a second
+  time. A file of at most 8 MiB is now parsed once and its records reused; larger files keep the two passes, so
+  memory stays bounded. On 300 files of 200 records, JSON reading took 37-38% less CPU, and a 100,000-record file
+  about 28% less end to end.
+
 ## v0.8.7
 
 - **The Iceberg catalog is opened once per run (#1101).** Every catalog call used to start a new REST session

{floe_python-0.8.7 → floe_python-0.8.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.7
+Version: 0.8.9
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.7 → floe_python-0.8.9}/context/architecture.md

@@ -93,8 +93,11 @@ them (see `context/decisions/polars-and-arrow.md`).
    archive inputs (D32) → promote state claims → write run.json
 ```
 
-`floe run` runs entities in config order. A policy abort fails only its entity; any raised
-`FloeError` aborts the run (D36). Reports are written under `<report.path>/run_<id>/`, or
+`floe run` runs entities in config order, or on `execution.orchestration.max_concurrent_entities`
+threads when `strategy` is `parallel` (`run/parallel.py`: never two on one Iceberg table, accepted
+path or state file, merges one at a time; reports in config order). A policy abort fails only its
+entity; any raised `FloeError` aborts the run (D36): no new entity starts, running ones finish, the
+first error in config order is raised. Reports are written under `<report.path>/run_<id>/`, or
 `<report.path>/<domain>/run_<id>/` for a single-domain run (D37).
 
 ## Key contracts

{floe_python-0.8.7 → floe_python-0.8.9}/docs/config.md

@@ -409,6 +409,7 @@ state file shape, and CAS behavior.
     - `max_size_per_file`: positive integer bytes (default: 256MB)
       - Parquet: split accepted parquet into parts at write time.
       - Iceberg: only sets the report's small-file threshold (half this value).
+    - Iceberg applies `compression` and `row_group_size` to `overwrite`/`append` data files only (not merges); see [Iceberg sink](sinks/iceberg.md).
     - Accepted rows are written once per entity, after every input file is read: Parquet
       parts in one pass, Iceberg in one catalog commit. Inputs are archived (when
       `sink.archive` is enabled) only after that write succeeds, so an input is never

{floe_python-0.8.7 → floe_python-0.8.9}/docs/profiles.md

@@ -31,6 +31,9 @@ metadata:                    # required
 execution:                   # optional
   runner:
     type: local              # required when execution is present
+  orchestration:             # optional; see "Running entities in parallel"
+    strategy: parallel       # optional; "sequential" (default) or "parallel"
+    max_concurrent_entities: 4  # optional; integer >= 1
 variables:                   # optional; flat string → string map
   KEY: value
 validation:                  # optional
@@ -355,6 +358,40 @@ Notes:
 - Full config validation (`floe validate -c config.yml --profile profiles/prod.yml`)
   additionally checks catalog/storage compatibility and entity sink bindings.
 
+## Running entities in parallel
+
+`execution.orchestration` also sets how many entities one `floe run` process runs at once
+(`floe run -c … -p …`, `floe run --manifest …` generated with the profile, and `floe.run(…, profile_path=…)`).
+Dagster reads the same fields to cap its own executor.
+
+```yaml
+execution:
+  runner: {type: local}
+  orchestration: {strategy: parallel, max_concurrent_entities: 4}
+```
+
+- **Opt-in.** Only `strategy: parallel` runs entities in parallel threads. `sequential`, no `orchestration`
+  block, or `max_concurrent_entities: 1` run them one after another in config order, as before.
+  `max_concurrent_entities` defaults to `min(4, CPU count)` when `strategy: parallel` leaves it unset.
+- **Never at the same time:** entities with the same Iceberg table (catalog, namespace, table), the same
+  accepted path or the same incremental state file; and `merge_scd1` / `merge_scd2` entities, which run
+  one at a time. They wait for each other while other entities proceed.
+- **Order.** `entity_started`, `file_*` and `entity_finished` events of different entities interleave, and
+  observers (including your `set_observer` callback) are called from worker threads. The
+  reports, the `run.summary.json` entity list and the console summary stay in config order.
+  Every entity's inputs are still listed before the first `entity_started`.
+- **Failure.** A policy `abort` fails only its entity. Any other error stops new entities from starting,
+  lets the running ones finish (they release or promote their incremental claims), then raises the first
+  error in config order. An interrupt (Ctrl-C) is handled the same way: the running entities finish first.
+- **Ephemeral disk.** Each running entity may hold up to its download prefetch window (8 files, about
+  512 MiB) in a temporary directory. Size ephemeral storage for `workers × 512 MiB` plus the largest file.
+- **Memory.** Each running entity holds its own chunk (up to 1M rows) in memory, and a merge entity's DuckDB
+  (`memory_limit` 4 GB) can run alongside non-merge entities. Peak memory grows with `max_concurrent_entities`:
+  size the pod for `workers` × one entity's working set (plus DuckDB when merges are configured), not for a sequential run.
+
+Existing profiles that already say `strategy: parallel` for Dagster now also run the entities of a
+`domain` Job in parallel inside its `floe run`.
+
 ## Hardening assessment and plan
 
 Current profile feature gaps and fixes:

{floe_python-0.8.7 → floe_python-0.8.9}/docs/sinks/iceberg.md

@@ -157,6 +157,11 @@ Current scope:
   run already committed and skips it (`skip_reason: already_committed`).
 - Unique checks on `append` read the table's existing keys, and a commit conflict re-checks the
   keys of concurrently added files.
+- `sink.accepted.options.compression` and `row_group_size` shape the Parquet data files of
+  `overwrite` and `append` writes (PyIceberg's `write.parquet.compression-codec` and
+  `write.parquet.row-group-limit`, applied to that write only; the table's properties are not
+  changed). Unset, the table's properties and PyIceberg's defaults apply. `merge_scd1` and
+  `merge_scd2` write through DuckDB and ignore both options.
 - `schema.schema_evolution.mode: add_columns` adds new columns as nullable (D13)
 - Unsigned integer columns are widened (`uint8`/`uint16` → `int`, `uint32` → `long`); Iceberg
   has no unsigned types
@@ -171,6 +176,7 @@ Current scope:
 ## Limitations / non-goals (current)
 
 - `strategy: drop` in `pii` is rejected: the table enforces the full declared schema
+- Every commit rewrites the table metadata, which lists every retained snapshot: an `overwrite` of a table with 100 snapshots took about 230 ms of commit and write time against about 50 ms with 2 on a local catalog. Run `floe maintain` to expire snapshots on tables written often
 - Snapshot expiry and orphan file removal with `floe maintain` (see [CLI usage](../cli.md)); compaction is experimental (`--compact --experimental`)
 
 ## Integration tests

{floe_python-0.8.7 → floe_python-0.8.9}/orchestrators/airflow-floe/example/config.sales.yml

@@ -20,8 +20,8 @@ entities:
         separator: ","
     sink:
       accepted:
-        format: "delta"
-        path: "./out/accepted/orders_delta"
+        format: "parquet"
+        path: "./out/accepted/orders"
       rejected:
         format: "csv"
         path: "./out/rejected/orders"

{floe_python-0.8.7 → floe_python-0.8.9}/orchestrators/airflow-floe/example/config.yml

@@ -52,8 +52,8 @@ entities:
         separator: ","
     sink:
       accepted:
-        format: "delta"
-        path: "./out/accepted/orders_delta"
+        format: "parquet"
+        path: "./out/accepted/orders"
       rejected:
         format: "csv"
         path: "./out/rejected/orders"

floe_python-0.8.9/orchestrators/airflow-floe/example/manifest.airflow.json

@@ -0,0 +1,243 @@
+{
+  "generated_at_ts_ms": 1793523660518,
+  "floe_version": "0.8.8",
+  "spec_version": "0.8",
+  "manifest_id": "mfv1-64b774849a18767a",
+  "manifest_revision": "sha256:b42d7a3d9d6904415060fe2f456927e6dd0f73bf0f21a1d76de1c90683626d7c",
+  "runtime_env": "cli",
+  "config_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/config.yml",
+  "config_checksum": "sha256:840cdb4eba82462ef12d8c1c0e2ed4360a60af634d23ed675683e2670f59b819",
+  "report_base_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/report",
+  "domains": [
+    {
+      "name": "hr",
+      "incoming_dir": "./in/hr"
+    },
+    {
+      "name": "sales",
+      "incoming_dir": "./in/sales"
+    }
+  ],
+  "execution": {
+    "entrypoint": "floe",
+    "base_args": [
+      "run",
+      "--manifest",
+      "{manifest_uri}",
+      "--log-format",
+      "json",
+      "--quiet"
+    ],
+    "per_entity_args": [
+      "--entities",
+      "{entity_name}"
+    ],
+    "log_format": "json",
+    "result_contract": {
+      "run_finished_event": true,
+      "summary_uri_field": "summary_uri",
+      "exit_codes": {
+        "0": "success_or_rejected",
+        "1": "technical_failure",
+        "2": "aborted"
+      }
+    },
+    "defaults": {
+      "env": {},
+      "workdir": null
+    }
+  },
+  "runners": {
+    "default": "local",
+    "definitions": {
+      "local": {
+        "type": "local_process",
+        "command": null,
+        "args": null,
+        "timeout_seconds": null,
+        "ttl_seconds_after_finished": null,
+        "poll_interval_seconds": null,
+        "secrets": null,
+        "image": null,
+        "namespace": null,
+        "service_account": null,
+        "resources": null,
+        "env": null,
+        "workspace_url": null,
+        "existing_cluster_id": null,
+        "config_uri": null,
+        "python_file_uri": null,
+        "job_name": null,
+        "auth": null,
+        "env_parameters": null
+      }
+    }
+  },
+  "entities": [
+    {
+      "name": "employees",
+      "domain": "hr",
+      "group_name": "hr",
+      "asset_key": [
+        "hr",
+        "employees"
+      ],
+      "source_format": "csv",
+      "accepted_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/employees",
+      "rejected_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/employees",
+      "tags": null,
+      "source": {
+        "format": "csv",
+        "storage": "local",
+        "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/in/hr/employees.csv",
+        "path": "./in/hr/employees.csv",
+        "resolved": true,
+        "cast_mode": "strict",
+        "options": {
+          "data_row": null,
+          "encoding": "UTF8",
+          "glob": null,
+          "header": true,
+          "header_row": 1,
+          "json_mode": null,
+          "namespace": null,
+          "null_values": [
+            ""
+          ],
+          "recursive": false,
+          "row_tag": null,
+          "separator": ",",
+          "sheet": null,
+          "value_tag": null
+        }
+      },
+      "sinks": {
+        "accepted": {
+          "format": "parquet",
+          "storage": "local",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/employees",
+          "path": "./out/accepted/employees",
+          "resolved": true
+        },
+        "rejected": {
+          "format": "csv",
+          "storage": "local",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/employees",
+          "path": "./out/rejected/employees",
+          "resolved": true
+        },
+        "archive": null
+      },
+      "runner": null,
+      "policy_severity": "reject",
+      "write_mode": "overwrite",
+      "incremental_mode": "none",
+      "schema": {
+        "columns": [
+          {
+            "name": "employee_id",
+            "column_type": "string",
+            "nullable": false,
+            "unique": true
+          },
+          {
+            "name": "start_date",
+            "column_type": "date",
+            "nullable": false
+          },
+          {
+            "name": "salary",
+            "column_type": "number",
+            "nullable": false
+          }
+        ],
+        "primary_key": [],
+        "unique_keys": []
+      }
+    },
+    {
+      "name": "orders",
+      "domain": "sales",
+      "group_name": "sales",
+      "asset_key": [
+        "sales",
+        "orders"
+      ],
+      "source_format": "csv",
+      "accepted_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/orders",
+      "rejected_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/orders",
+      "tags": null,
+      "source": {
+        "format": "csv",
+        "storage": "local",
+        "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/in/sales/orders.csv",
+        "path": "./in/sales/orders.csv",
+        "resolved": true,
+        "cast_mode": "strict",
+        "options": {
+          "data_row": null,
+          "encoding": "UTF8",
+          "glob": null,
+          "header": true,
+          "header_row": 1,
+          "json_mode": null,
+          "namespace": null,
+          "null_values": [
+            ""
+          ],
+          "recursive": false,
+          "row_tag": null,
+          "separator": ",",
+          "sheet": null,
+          "value_tag": null
+        }
+      },
+      "sinks": {
+        "accepted": {
+          "format": "parquet",
+          "storage": "local",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/orders",
+          "path": "./out/accepted/orders",
+          "resolved": true
+        },
+        "rejected": {
+          "format": "csv",
+          "storage": "local",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/orders",
+          "path": "./out/rejected/orders",
+          "resolved": true
+        },
+        "archive": null
+      },
+      "runner": null,
+      "policy_severity": "reject",
+      "write_mode": "overwrite",
+      "incremental_mode": "none",
+      "schema": {
+        "columns": [
+          {
+            "name": "order_id",
+            "column_type": "string",
+            "nullable": false,
+            "unique": true
+          },
+          {
+            "name": "created_at",
+            "column_type": "datetime",
+            "nullable": false
+          },
+          {
+            "name": "amount",
+            "column_type": "number",
+            "nullable": false
+          }
+        ],
+        "primary_key": [],
+        "unique_keys": []
+      }
+    }
+  ],
+  "metadata": {
+    "project": "floe-airflow-example"
+  }
+}

{floe_python-0.8.7 → floe_python-0.8.9}/orchestrators/airflow-floe/example/manifests/hr.manifest.json

@@ -1,12 +1,13 @@
 {
-  "schema": "floe.manifest.v1",
-  "generated_at_ts_ms": 1771245437766,
-  "floe_version": "0.2.4",
-  "spec_version": "0.2",
-  "manifest_id": "mfv1-1382cbe0a11e3ae3",
-  "config_uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/config.hr.yml",
-  "config_checksum": null,
-  "report_base_uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./report",
+  "generated_at_ts_ms": 1793523661633,
+  "floe_version": "0.8.8",
+  "spec_version": "0.8",
+  "manifest_id": "mfv1-64876c0cce7b927c",
+  "manifest_revision": "sha256:056899ce3fb909b7e3cab9af4e00460151ea4d3a76d4faf8302ce13f048976a1",
+  "runtime_env": "cli",
+  "config_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/config.hr.yml",
+  "config_checksum": "sha256:01a9497a45f2a9099a44db4e979a364916565d9ea84629327685f272e7a19082",
+  "report_base_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/report",
   "domains": [
     {
       "name": "hr",
@@ -17,8 +18,8 @@
     "entrypoint": "floe",
     "base_args": [
       "run",
-      "-c",
-      "{config_uri}",
+      "--manifest",
+      "{manifest_uri}",
       "--log-format",
       "json",
       "--quiet"
@@ -32,8 +33,8 @@
       "run_finished_event": true,
       "summary_uri_field": "summary_uri",
       "exit_codes": {
-        "1": "technical_failure",
         "0": "success_or_rejected",
+        "1": "technical_failure",
         "2": "aborted"
       }
     },
@@ -47,11 +48,24 @@
     "definitions": {
       "local": {
         "type": "local_process",
+        "command": null,
+        "args": null,
+        "timeout_seconds": null,
+        "ttl_seconds_after_finished": null,
+        "poll_interval_seconds": null,
+        "secrets": null,
         "image": null,
         "namespace": null,
         "service_account": null,
         "resources": null,
-        "env": null
+        "env": null,
+        "workspace_url": null,
+        "existing_cluster_id": null,
+        "config_uri": null,
+        "python_file_uri": null,
+        "job_name": null,
+        "auth": null,
+        "env_parameters": null
       }
     }
   },
@@ -65,13 +79,13 @@
         "employees"
       ],
       "source_format": "csv",
-      "accepted_sink_uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./out/accepted/employees",
-      "rejected_sink_uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./out/rejected/employees",
+      "accepted_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/employees",
+      "rejected_sink_uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/employees",
       "tags": null,
       "source": {
         "format": "csv",
         "storage": "local",
-        "uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./in/hr/employees.csv",
+        "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/in/hr/employees.csv",
         "path": "./in/hr/employees.csv",
         "resolved": true,
         "cast_mode": "strict",
@@ -97,14 +111,14 @@
         "accepted": {
           "format": "parquet",
           "storage": "local",
-          "uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./out/accepted/employees",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/accepted/employees",
           "path": "./out/accepted/employees",
           "resolved": true
         },
         "rejected": {
           "format": "csv",
           "storage": "local",
-          "uri": "local:///Users/a.metwalli/floe/orchestrators/airflow-floe/example/./out/rejected/employees",
+          "uri": "local:///home/user/floe/orchestrators/airflow-floe/example/out/rejected/employees",
           "path": "./out/rejected/employees",
           "resolved": true
         },
@@ -137,5 +151,8 @@
         "unique_keys": []
       }
     }
-  ]
-}
+  ],
+  "metadata": {
+    "project": "floe-airflow-example-hr"
+  }
+}