floe-python 0.8.6__tar.gz → 0.8.8__tar.gz

{floe_python-0.8.6 → floe_python-0.8.8}/CHANGELOG.md

@@ -2,6 +2,55 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.8
+
+- **Entities can run in parallel in one `floe run` (#1105).** With `execution.orchestration.strategy: parallel` in
+  the profile, `floe run -c … -p …`, `floe run --manifest …` and `floe.run(…, profile_path=…)` run up to
+  `max_concurrent_entities` entities at once (default `min(4, CPU count)`). Entities that share an Iceberg table, an
+  accepted path or a state file never run together, and merge entities run one at a time. Reports and the summary
+  stay in config order; events of different entities interleave and observers are called from worker threads. On
+  the Polaris/SeaweedFS harness, 12 entities writing 12 Iceberg tables took 2.04 s with one worker and 1.03 s with
+  four. Without `strategy: parallel` nothing changes. Profiles that already set `strategy: parallel` for Dagster now
+  also run the entities of a `job_scope: domain` Job in parallel; peak memory then grows with the number of workers
+  (see `docs/profiles.md`).
+
+- **Creating a new Iceberg namespace no longer fails when two writers race (#1117).** Two entities, or two
+  processes such as concurrent Dagster Jobs, that both found a namespace missing and created it made the second
+  creation fail the run. A namespace that already exists by then now counts as created.
+
+- **A small JSON file is parsed once (#1115).** The key check before the read parsed every JSON file a second
+  time. A file of at most 8 MiB is now parsed once and its records reused; larger files keep the two passes, so
+  memory stays bounded. On 300 files of 200 records, JSON reading took 37-38% less CPU, and a 100,000-record file
+  about 28% less end to end.
+
+## v0.8.7
+
+- **The Iceberg catalog is opened once per run (#1101).** Every catalog call used to start a new REST session
+  (a config request, plus an OAuth token request when a credential is set), and an `append` entity did it twice.
+  A run now reuses one catalog per catalog definition and storage, so N entities no longer pay up to 2N
+  initialisations. The catalog is opened again at the start of each run.
+
+- **S3 clients are reused within a run (#1100).** Each `S3Client` builds its own boto3 session, about 55-120 ms,
+  and one S3 entity built 6-8 of them (source, state, accepted, rejected, archive, report). A run now builds one
+  client per storage definition: a two-entity run on one storage went from 5 clients to 1. `AWS_*` variables and
+  credentials are read again at the start of each run, and by `inspect_entity_state()` and
+  `reset_entity_state()`.
+
+- **Remote inputs are downloaded ahead of the file being read (#1102).** Each pending S3 file used to be
+  downloaded only when its turn came, so the read loop waited on every GET. An entity now downloads up to 8
+  files, and about 512 MiB, ahead of the one it is validating (a single larger file still downloads, alone).
+  Files are still processed in the same order, a failed download still raises its own error when that file's turn
+  comes, and local sources are unchanged. With a simulated 0.1 s download per file, 16 files went from 1.72 s to
+  0.24 s.
+
+- **Small CSV files cost fewer Polars plans (#1104).** A clean 200-row file ran 16 query plans between the
+  reader and the writer and now runs 6, with identical output. On 1,200 such files (8 entities of 150 files,
+  Parquet sink on local disk) the run was about 26-28% faster, 7.45 s to 5.38 s in the last measurement.
+
+- **`poll_interval_seconds` and `job_scope` are documented (#1103).** `docs/profiles.md` and
+  `context/orchestrators.md` describe the Kubernetes runner's status poll (default 10 s, a Job that finishes
+  early is noticed at the next poll) and when to choose `job_scope: domain` over `entity`.
+
 ## v0.8.6
 
 - **Merge modes keep a key's latest version when it repeats across pending files (#1090).** With `merge_scd1`
@@ -63,18 +112,19 @@ All notable changes to Floe are documented in this file.
 
 ## Unreleased
 
-### dagster-floe
-
-- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
-  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
-  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
-  `merge_scd1` / `merge_scd2` entity needs a Floe version that accepts `--full-refresh` with merge modes.
-
 ### airflow-floe
 
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
   dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
+## dagster-floe v0.4.3
+
+- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
+  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
+  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
+  `merge_scd1` entity needs Floe 0.8.6 or newer, which rebuilds the table from the inputs; `merge_scd2`
+  still rejects `--full-refresh`.
+
 ## dagster-floe v0.4.2
 
 - **Silver lineage events from a `kubernetes_job` run now carry the Dagster parent run (#1076).** The Job

{floe_python-0.8.6 → floe_python-0.8.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.6
+Version: 0.8.8
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.6 → floe_python-0.8.8}/context/architecture.md

@@ -93,8 +93,11 @@ them (see `context/decisions/polars-and-arrow.md`).
    archive inputs (D32) → promote state claims → write run.json
 ```
 
-`floe run` runs entities in config order. A policy abort fails only its entity; any raised
-`FloeError` aborts the run (D36). Reports are written under `<report.path>/run_<id>/`, or
+`floe run` runs entities in config order, or on `execution.orchestration.max_concurrent_entities`
+threads when `strategy` is `parallel` (`run/parallel.py`: never two on one Iceberg table, accepted
+path or state file, merges one at a time; reports in config order). A policy abort fails only its
+entity; any raised `FloeError` aborts the run (D36): no new entity starts, running ones finish, the
+first error in config order is raised. Reports are written under `<report.path>/run_<id>/`, or
 `<report.path>/<domain>/run_<id>/` for a single-domain run (D37).
 
 ## Key contracts

{floe_python-0.8.6 → floe_python-0.8.8}/context/orchestrators.md

@@ -211,6 +211,7 @@ When `type = kubernetes_job`, the manifest runner definition supports:
 | `node_selector` | object | Pod `nodeSelector` |
 | `tolerations` | array | Pod `tolerations`, verbatim |
 | `backoff_limit` | integer >= 0 | Job `backoffLimit` (default `0`) |
+| `poll_interval_seconds` | integer >= 1 | Seconds between status reads of the Job (default `10`) |
 | `env_from` | array | Container `envFrom`, verbatim, e.g. `[{secretRef: {name: floe-env}}]` |
 
 These fields are populated automatically when `floe manifest generate` is given a profile with a `kubernetes_job` runner block. The `dagster-floe[kubernetes]` optional extra expresses the Kubernetes client requirement.
@@ -220,6 +221,8 @@ These fields are populated automatically when `floe manifest generate` is given
 - **`entity`** (default, also when unset): one Job per entity, named `floe-<entity>-<id>`. Pod start is paid once per entity; each entity has its own timeout, TTL and failure. Use it for a few large entities, or when entities need isolation.
 - **`domain`**: entities of one domain whose runner has `job_scope: domain` form one subsettable multi-asset per (domain, runner), so an entity with another runner gets its own Job. Materializing it launches **one** Job named `floe-<domain>-<id>` running `floe run --manifest <uri> --entities a,b,c` (`--domain <d>` when they are every entity of the domain in the manifest) for the selected entities in manifest order, one after another, so Polaris never sees overlapping commits from the Job. The Job is sized from `domain_resources[<domain>]`, else `resources`; `timeout_seconds`, TTL and polling apply to the one Job. Use it when pod start dominates (many small entities). Select the accepted assets: a rejected asset is materialized only when it is selected too, and the asset job selects both. The `floe/table` op tag (#924) is not set on a domain asset.
 
+**Choosing a scope and a poll interval.** The runner reads the Job's status every `poll_interval_seconds`, so a Job that finishes early is only noticed at the next poll: with the default 10 s, a 3 s Job is seen about 7 s late, a flat per-Job cost. `entity` pays pod start and that polling delay once per entity, but entities are isolated and run in parallel through Dagster's executor. `domain` pays both once per domain, but runs its entities one after another. For short Jobs, lower `poll_interval_seconds` (for example `2`), at the cost of more Kubernetes API reads.
+
 Dagster still reports per entity (materialization, rows, rejected counts, report URI, asset checks) from the run's `run_finished` event and summary.
 
 **After a failure** each entity is reported from its own status in the run summary: `rejected` and `success_with_warnings` entities are materialized, an `aborted` / `failed` entity fails, and an entity absent from the summary never ran and is reported as *not run* (listed in the step failure's `not_run_entities`), not as its own failure. Since D36 (#959) a policy `severity: abort` fails only its entity and the run continues with the next one. A global failure (access, memory, any raised `FloeError`) still aborts the run. Re-executing the step reruns the selected entities; subset it to rerun only some. Report URIs come from `run_finished` (`summary_uri`, `entity_report_uris`), never built by hand (#960).

{floe_python-0.8.6 → floe_python-0.8.8}/docs/profiles.md

@@ -31,6 +31,9 @@ metadata:                    # required
 execution:                   # optional
   runner:
     type: local              # required when execution is present
+  orchestration:             # optional; see "Running entities in parallel"
+    strategy: parallel       # optional; "sequential" (default) or "parallel"
+    max_concurrent_entities: 4  # optional; integer >= 1
 variables:                   # optional; flat string → string map
   KEY: value
 validation:                  # optional
@@ -238,12 +241,22 @@ execution:
     node_selector: {pool: batch}
     tolerations: [{key: batch, operator: Exists, effect: NoSchedule}]   # verbatim
     backoff_limit: 2                         # Job backoffLimit, default 0
+    poll_interval_seconds: 2                 # seconds between Job status reads, default 10
     env_from: [{secretRef: {name: floe-env}}]   # container envFrom, verbatim
 ```
 
 `labels`, `annotations` and `node_selector` are string maps, `backoff_limit` is a non-negative integer,
 and `security_context`, `tolerations` and `env_from` are copied as written: Kubernetes validates them.
 
+`poll_interval_seconds` is an integer of at least 1 (default 10): dagster-floe reads the Job's status that
+often, so a Job that finishes early is only noticed at the next poll. A lower value such as `2` shortens
+short Jobs at the cost of more Kubernetes API reads.
+
+`job_scope` (`entity` by default, or `domain`) sets how many Jobs a run launches. `entity` starts one Job
+per entity: each pays pod start and one poll interval, but they are isolated and run in parallel through
+Dagster's executor. `domain` starts one Job per domain: pod start and polling are paid once, but its
+entities run one after another.
+
 ---
 
 ## Using profile variables in a Floe config
@@ -345,6 +358,40 @@ Notes:
 - Full config validation (`floe validate -c config.yml --profile profiles/prod.yml`)
   additionally checks catalog/storage compatibility and entity sink bindings.
 
+## Running entities in parallel
+
+`execution.orchestration` also sets how many entities one `floe run` process runs at once
+(`floe run -c … -p …`, `floe run --manifest …` generated with the profile, and `floe.run(…, profile_path=…)`).
+Dagster reads the same fields to cap its own executor.
+
+```yaml
+execution:
+  runner: {type: local}
+  orchestration: {strategy: parallel, max_concurrent_entities: 4}
+```
+
+- **Opt-in.** Only `strategy: parallel` runs entities in parallel threads. `sequential`, no `orchestration`
+  block, or `max_concurrent_entities: 1` run them one after another in config order, as before.
+  `max_concurrent_entities` defaults to `min(4, CPU count)` when `strategy: parallel` leaves it unset.
+- **Never at the same time:** entities with the same Iceberg table (catalog, namespace, table), the same
+  accepted path or the same incremental state file; and `merge_scd1` / `merge_scd2` entities, which run
+  one at a time. They wait for each other while other entities proceed.
+- **Order.** `entity_started`, `file_*` and `entity_finished` events of different entities interleave, and
+  observers (including your `set_observer` callback) are called from worker threads. The
+  reports, the `run.summary.json` entity list and the console summary stay in config order.
+  Every entity's inputs are still listed before the first `entity_started`.
+- **Failure.** A policy `abort` fails only its entity. Any other error stops new entities from starting,
+  lets the running ones finish (they release or promote their incremental claims), then raises the first
+  error in config order. An interrupt (Ctrl-C) is handled the same way: the running entities finish first.
+- **Ephemeral disk.** Each running entity may hold up to its download prefetch window (8 files, about
+  512 MiB) in a temporary directory. Size ephemeral storage for `workers × 512 MiB` plus the largest file.
+- **Memory.** Each running entity holds its own chunk (up to 1M rows) in memory, and a merge entity's DuckDB
+  (`memory_limit` 4 GB) can run alongside non-merge entities. Peak memory grows with `max_concurrent_entities`:
+  size the pod for `workers` × one entity's working set (plus DuckDB when merges are configured), not for a sequential run.
+
+Existing profiles that already say `strategy: parallel` for Dagster now also run the entities of a
+`domain` Job in parallel inside its `floe run`.
+
 ## Hardening assessment and plan
 
 Current profile feature gaps and fixes:

{floe_python-0.8.6 → floe_python-0.8.8}/orchestrators/dagster-floe/README.md

@@ -42,6 +42,7 @@ pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.3,<0.9 for t
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.3 | 0.8.3 or newer (the `floe/full_refresh` run tag on a `merge_scd1` entity needs 0.8.6) | `0.8` |
 | 0.4.2 | 0.8.3 or newer | `0.8` |
 | 0.4.1 | 0.8.3 or newer (`kubernetes_job` runs pass the profile as `-p env:FLOE_PROFILE_CONTENT`) | `0.8` |
 | 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |

{floe_python-0.8.6 → floe_python-0.8.8}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.4.2"
+version = "0.4.3"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"

{floe_python-0.8.6 → floe_python-0.8.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.6"
+version = "0.8.8"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.6 → floe_python-0.8.8}/scripts/test_config_schema.py

@@ -106,7 +106,7 @@ class ConfigSchemaTest(unittest.TestCase):
             types.SchemaEvolutionConfig: "schema_evolution", types.SchemaMismatchConfig: "schema_mismatch",
             types.ColumnConfig: "column",
         }  # `delta` and `duckdb` stay raw in the model (D1) and have no binding.
-        internal = {project.RootConfig: {"config_files"}, project.DomainConfig: {"resolved_incoming_dir"}, types.EntityConfig: {"id"}, types.SinkTarget: {"write_mode"}}
+        internal = {project.RootConfig: {"config_files", "entity_workers"}, project.DomainConfig: {"resolved_incoming_dir"}, types.EntityConfig: {"id"}, types.SinkTarget: {"write_mode"}}
         aliases = {types.SourceConfig: {"source_ref": "ref"}, types.ColumnConfig: {"column_type": "type"}}
         wire_extras = {
             project.RootConfig: {"sources", "include", "filesystems"},

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/__init__.py

@@ -46,6 +46,7 @@ from floe.run.events import (
 )
 from floe.state import dump_state, read_state, reset
 from floe.storage.paths import normalize_local_path
+from floe.storage.resolve import clear_storage_clients
 from floe.storage.types import StorageClient
 
 _observer: Callable[[RunEvent], None] | None = None
@@ -135,8 +136,7 @@ def _profile(
         profile = parse_profile(path)
     except FloeError as e:
         raise FloeConfigError(f"failed to load profile {_debug(path)}: {e}") from e
-    return ProfileConfig(profile.name, profile.runner, profile.variables | (variables or {}),
-                         profile.catalogs, profile.storages, profile.lineage)  # fmt: skip
+    return replace(profile, variables=profile.variables | (variables or {}))
 
 
 def _load(config_path: str, profile: ProfileConfig | None = None) -> tuple[Path, _Root]:
@@ -281,6 +281,7 @@ def extract_config_env_vars(config_path: str) -> dict[str, str]:
 
 
 def _state(config_path: str, entity_name: str) -> tuple[str, str, str, StorageClient]:
+    clear_storage_clients()
     path, root = _load(config_path)
     [entity] = _run.select_entities(root, [entity_name])
     uri, client = state_uri(entity, normalize_local_path(path.parent), root.storages)

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/_cli_output.py

@@ -1,6 +1,7 @@
 """`floe run` output: run events as log lines and the run summary (Rust: `crates/floe-cli/src/logging.rs`, `output.rs`)."""
 
 import json
+import threading
 import time
 from typing import Any
 
@@ -73,18 +74,20 @@ class Console:
     def __init__(self, log_format: str, run_id: str) -> None:
         self.log_format, self.run_id = log_format, run_id
         self.started = False
-        self.timings: list[dict[str, int]] = []  # per entity: input -> elapsed_ms
+        self.timings: dict[
+            str, dict[str, int]
+        ] = {}  # entity name -> input -> elapsed_ms
+        self._lock = threading.Lock()  # entities may run in parallel threads
 
     def __call__(self, event: dict[str, Any]) -> None:
         kind = event["event"]
-        self.started |= kind == "run_started"
-        if kind == "entity_started":
-            self.timings.append({})
-        elif kind == "file_finished" and self.timings:
-            self.timings[-1][event["input"]] = event["elapsed_ms"]
-        if self.log_format != "off":
-            fmt = event_json if self.log_format == "json" else event_text
-            print(fmt(event), flush=True)
+        with self._lock:
+            self.started |= kind == "run_started"
+            if kind == "file_finished":
+                self.timings.setdefault(event["entity"], {})[event["input"]] = event["elapsed_ms"]  # fmt: skip
+            if self.log_format != "off":
+                fmt = event_json if self.log_format == "json" else event_text
+                print(fmt(event), flush=True)
 
     def failed(self, err: FloeError, config: str | None) -> None:
         """Rust's `emit_failed_run_events`, after a `run_started` for `config` when the run never started.
@@ -154,7 +157,7 @@ def run_output(
     outcome: RunOutcome,
     entities: list[EntityConfig],
     mode: str,
-    timings: list[dict[str, int]],
+    timings: dict[str, dict[str, int]],
 ) -> str:
     """Rust's `format_run_output`; `mode` is `default`, `quiet` or `verbose`."""
     run_dir = outcome.run_dir  # a URI on remote storage: join with `/`, never `Path`
@@ -166,7 +169,17 @@ def run_output(
         results = dict.fromkeys(("files_total", "rows_total", "accepted_total", "rejected_total", "files_skipped"), 0)  # fmt: skip
         status, exit_code = "success", 0
     else:
-        blocks = [_entity(r, t, mode == "verbose") for r, t in zip(outcome.entity_reports, timings, strict=True)]  # fmt: skip
+        names = {
+            e.id: e.name for e in entities
+        }  # reports carry the id, `file_finished` events the name
+
+        def shown(report: dict[str, Any]) -> list[str]:
+            ident: str = report["entity"]["name"]
+            return _entity(
+                report, timings.get(names.get(ident, ident), {}), mode == "verbose"
+            )
+
+        blocks = [shown(r) for r in outcome.entity_reports]
         results, run = outcome.summary["results"], outcome.summary["run"]
         status, exit_code = run["status"], run["exit_code"]
     lines = ["DRY RUN MODE - No actual execution performed"] if outcome.dry_run is not None else []  # fmt: skip

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/checks/mismatch.py

@@ -119,19 +119,34 @@ def apply_mismatch(df: pl.DataFrame, plan: MismatchOutcome) -> pl.DataFrame:
     return df
 
 
-def json_input_columns(path: Path, mode: str = "array") -> list[str] | None:
-    """Sorted top-level keys of every record, read exactly as `read_json` reads them.
+def json_probe(
+    path: Path, mode: str, limit: int
+) -> tuple[list[str] | None, list[tuple[str, Any]] | None]:
+    """`json_input_columns` plus the file's parsed records when it has at most `limit` (else `None`).
 
-    D40: the union over all records, so a key only a later record carries is still an input
-    column (Rust 0.7 probed the first object only). `None` for a file without records (D41), which
-    differs from records that carry no keys (`[]`): those are still checked for missing columns.
+    A file that fits one chunk is then parsed once: the caller reads the records it got here.
     """
     if mode not in ("array", "ndjson"):
         raise FloeIoError(
             f"unsupported source.options.json_mode={mode} (allowed: array, ndjson)"
         )
-    # ponytail: the file is parsed twice (probe, then read); fold the probe into the read if it costs
     keys: set[str] | None = None
-    for _, record in json_records(path, mode):
-        keys = (keys or set[str]()) | {str(k) for k in record}
-    return None if keys is None else sorted(keys)
+    head: list[tuple[str, Any]] | None = []
+    for located in json_records(path, mode):
+        keys = keys or set[str]()
+        keys.update(str(k) for k in located[1])
+        if head is not None and len(head) < limit:
+            head.append(located)
+        else:  # a longer file is read again, chunk by chunk, so it keeps nothing
+            head = None
+    return (None if keys is None else sorted(keys)), head
+
+
+def json_input_columns(path: Path, mode: str = "array") -> list[str] | None:
+    """Sorted top-level keys of every record, read exactly as `read_json` reads them.
+
+    D40: the union over all records, so a key only a later record carries is still an input
+    column (Rust 0.7 probed the first object only). `None` for a file without records (D41), which
+    differs from records that carry no keys (`[]`): those are still checked for missing columns.
+    """
+    return json_probe(path, mode, 0)[0]

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/checks/normalize.py

@@ -115,4 +115,6 @@ def normalize_columns(df: pl.DataFrame, strategy: str | None) -> pl.DataFrame:
                 f"normalized input column collision: {seen[new]} and {old} -> {new}"
             )
         seen[new] = old
-    return df.rename(names)
+    renamed = df.clone()
+    renamed.columns = list(names.values())  # a rename is a query plan, this is not
+    return renamed

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/checks/rows.py

@@ -109,6 +109,10 @@ def split_rows(
     sources: Mapping[str, str] | None = None,
 ) -> tuple[pl.DataFrame, pl.DataFrame]:
     """`(accepted, rejected)`; rejected adds `__floe_row_index` (UInt64, row position in `df`) and `__floe_errors`."""
+    # a null mask passes the row, as in `error_column`
+    if not any(c.mask.any() for c in checks):
+        none = [pl.Series(ROW_INDEX, [], pl.UInt64), pl.Series(ERRORS, [], pl.String)]
+        return df, df.clear().with_columns(none)  # replaces a clashing column, as below
     errors = error_column(df.height, checks, formatter, sources)
     rejected = df.with_columns(
         pl.int_range(pl.len(), dtype=pl.UInt64).alias(ROW_INDEX), errors

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/config/profile.py

@@ -1,7 +1,7 @@
 """Environment profiles, ported from `profile/parse.rs` and `profile/validate.rs`."""
 
 import os
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
 from dataclasses import dataclass
 from pathlib import Path
 from typing import cast
@@ -395,12 +395,21 @@ def validate_profile(profile: ProfileConfig) -> None:
         check_lineage(profile.lineage, "profile.lineage")
 
 
+def entity_workers(orchestration: Mapping[str, object] | None) -> int:
+    """How many entities a run may run at once: 1 unless `strategy` is `parallel`, then `max_concurrent_entities` or `min(4, cpus)`."""
+    if not orchestration or orchestration.get("strategy") != "parallel":
+        return 1
+    limit = orchestration.get("max_concurrent_entities")
+    return limit if isinstance(limit, int) else min(4, os.cpu_count() or 1)
+
+
 def apply_profile(config: RootConfig, profile: ProfileConfig) -> RootConfig:
     """The profile's blocks replace the config's, as `apply_profile_catalogs`, `_storages` and `_lineage` do."""
     blocks = {
         "catalogs": profile.catalogs,
         "storages": profile.storages,
         "lineage": profile.lineage,
+        "entity_workers": entity_workers(profile.orchestration),
     }
     return config.model_copy(update={k: v for k, v in blocks.items() if v is not None})
 

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/config/project.py

@@ -93,6 +93,7 @@ class RootConfig(BaseModel):
     lineage: LineageConfig | None
     entities: list[EntityConfig]
     config_files: list[Path]
+    entity_workers: int = 1  # profile/manifest orchestration; not a config key
 
 
 @dataclass

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/iceberg/catalog.py

@@ -4,13 +4,15 @@ import os
 import re
 import shutil
 import tempfile
+import threading
 import weakref
 from collections.abc import Generator, Sequence
-from contextlib import contextmanager
+from contextlib import contextmanager, suppress
 from urllib.parse import urlsplit
 
 import duckdb
 from pyiceberg.catalog import Catalog, load_catalog
+from pyiceberg.exceptions import NamespaceAlreadyExistsError
 from pyiceberg.partitioning import PartitionField, PartitionSpec
 from pyiceberg.schema import Schema
 from pyiceberg.table import Table
@@ -83,8 +85,34 @@ def _credential_props(catalog: str, credential: str) -> dict[str, str]:
     return {"credential": credential.removeprefix("client_credentials:")}
 
 
+_CATALOGS: dict[tuple[CatalogDefinition, StorageDefinition | None], Catalog] = {}
+_CATALOGS_LOCK = threading.Lock()
+
+
+def clear_catalogs() -> None:
+    """Called when a run starts: a catalog is reused within a run only, never across runs (#1101)."""
+    with _CATALOGS_LOCK:
+        _CATALOGS.clear()
+
+
 def open_catalog(
     definition: CatalogDefinition, storage: StorageDefinition | None = None
+) -> Catalog:
+    """One catalog per definition and storage until `clear_catalogs`, so the REST init (config, OAuth token) happens once.
+
+    Safe to call from several threads (one init per key). PyIceberg 0.12 refetches an expired OAuth token
+    itself: its REST calls retry once on a 401 after a new client-credentials request; a static `token:` has
+    nothing to refetch, as with a freshly opened catalog. A failed init is not kept.
+    """
+    key = (definition, storage)
+    with _CATALOGS_LOCK:
+        if key not in _CATALOGS:
+            _CATALOGS[key] = _open_catalog(definition, storage)
+        return _CATALOGS[key]
+
+
+def _open_catalog(
+    definition: CatalogDefinition, storage: StorageDefinition | None
 ) -> Catalog:
     """Credentials for S3 and Glue come from the environment; the catalog never vends them (as in Rust)."""
     props = {
@@ -323,7 +351,10 @@ def ensure_table(
     if not namespace_exists:
         if not create_namespace:
             raise FloeRunError(f"entity.name={entity} iceberg namespace {namespace} does not exist and create_database_if_missing is false")  # fmt: skip
-        with run_error("rest catalog namespace create failed"):
+        with (
+            run_error("rest catalog namespace create failed"),
+            suppress(NamespaceAlreadyExistsError),
+        ):  # another entity or process created it first
             catalog.create_namespace(ident)
     identifier = (namespace, name)
     with run_error("rest catalog table_exists check failed"):

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/lineage/observer.py

@@ -110,7 +110,6 @@ def _warn(message: str) -> None:
 class OpenLineageObserver:
     """A run observer (`floe.run.events.Observer`) posting OpenLineage `RunEvent`s."""
 
-    # ponytail: unsynchronised breaker counters; add a lock when entities run in parallel.
     def __init__(
         self, config: LineageConfig, entities: Sequence[EntityConfig], config_path: str
     ) -> None:
@@ -127,12 +126,14 @@ class OpenLineageObserver:
         self.run_uuid: str | None = None
         self.consecutive_failures = 0
         self.circuit_open = False
+        self._lock = threading.Lock()  # entities may run in parallel threads
 
     def __call__(self, event: dict[str, Any]) -> None:
         kind, ts = event["event"], event["ts_ms"]
         job = self.events.run_job_name
         if kind == "run_started":
-            self.consecutive_failures, self.circuit_open = 0, False
+            with self._lock:
+                self.consecutive_failures, self.circuit_open = 0, False
             self.run_uuid = str(uuid.uuid4())
             self._post("START", ts, self.run_uuid, job)
         elif kind == "entity_started":
@@ -171,19 +172,19 @@ class OpenLineageObserver:
             time.sleep(delay)
             retryable = self._attempt(data)
             if retryable is None:
-                self.consecutive_failures = 0
+                with self._lock:
+                    self.consecutive_failures = 0
                 return
             if not retryable:
                 break
         _warn(f"POST {self.url} failed — event dropped")
-        self.consecutive_failures += 1
-        failures = self.consecutive_failures
-        if (
-            failures
-            >= (3 if self.config.max_failures is None else self.config.max_failures)
-            and not self.circuit_open
-        ):
-            self.circuit_open = True
+        with self._lock:
+            self.consecutive_failures += 1
+            failures = self.consecutive_failures
+            limit = 3 if self.config.max_failures is None else self.config.max_failures
+            opening = failures >= limit and not self.circuit_open
+            self.circuit_open |= opening
+        if opening:
             _warn(f"disabled for this run after {failures} consecutive failures — check endpoint {self.url}")  # fmt: skip
 
     def _attempt(self, data: bytes) -> bool | None:
@@ -251,6 +252,7 @@ def run_with_lineage(
         return run(
             path, root, run_id=run_id, entities=entities, dry_run=dry_run,
             full_refresh=full_refresh, observer=observe, config_uri=config_uri,
+            workers=root.entity_workers,
         )  # fmt: skip
     except FloeError:
         if lineage:

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/manifest_run.py

@@ -12,6 +12,7 @@ from typing import Any, TypeVar, cast, get_args
 
 from pydantic import BaseModel, ValidationError
 
+from floe.config.profile import entity_workers
 from floe.config.project import (
     LineageConfig,
     ProjectMetadata,
@@ -265,6 +266,9 @@ def load_manifest(location: str | Path) -> tuple[RootConfig, list[EntityConfig],
         spec = m["spec_version"]
         entities = [_entity(e, _named_local(m)) for e in m["entities"]]
         report_base = m["report_base_uri"]
+        workers = entity_workers(
+            cast(Json, m.get("execution") or {}).get("orchestration")
+        )
         meta = m.get("metadata")  # D20: dumped without its None fields
         metadata = None if meta is None else ProjectMetadata.model_validate(_omitted_as_none(ProjectMetadata, meta))  # fmt: skip
     except (ValueError, KeyError, TypeError, AttributeError) as e:
@@ -289,6 +293,7 @@ def load_manifest(location: str | Path) -> tuple[RootConfig, list[EntityConfig],
         lineage=_section(LineageConfig, m, "lineage"),
         entities=entities,
         config_files=[],
+        entity_workers=workers,
     )
     observed = [_lineage_entity(e, me) for e, me in zip(entities, m["entities"])]
     return root, observed, _replay_base(location, m)

{floe_python-0.8.6 → floe_python-0.8.8}/src/floe/read/csv.py

@@ -79,15 +79,16 @@ def _scan_args(options: SourceOptions) -> _Args:
 
 def _columns(path: Path, args: _Args, declared: Sequence[str]) -> list[str]:
     try:
-        probe = args.scan(path, n_rows=0 if args.has_header else 1).collect()
+        probe = args.scan(path, n_rows=0 if args.has_header else 1)
+        if args.has_header:  # the schema is the header: no plan to run
+            return probe.collect_schema().names()
+        width = probe.collect().width
     except (pl.exceptions.PolarsError, OSError) as err:
         # Rust's polars prints the error kind: `no data: empty CSV`
         kind = "no data: " if isinstance(err, pl.exceptions.NoDataError) else ""
         raise FloeIoError(f"csv header read failed: {kind}{err}") from err
-    if args.has_header:
-        return probe.columns
-    names = list(declared[: probe.width])
-    return names + [f"extra_column_{i + 1}" for i in range(len(declared), probe.width)]
+    names = list(declared[:width])
+    return names + [f"extra_column_{i + 1}" for i in range(len(declared), width)]
 
 
 def _field_counts(path: Path, args: _Args, skip: int) -> Iterator[int]: