floe-python 0.8.7__tar.gz → 0.8.8__tar.gz

{floe_python-0.8.7 → floe_python-0.8.8}/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.8
+
+- **Entities can run in parallel in one `floe run` (#1105).** With `execution.orchestration.strategy: parallel` in
+  the profile, `floe run -c … -p …`, `floe run --manifest …` and `floe.run(…, profile_path=…)` run up to
+  `max_concurrent_entities` entities at once (default `min(4, CPU count)`). Entities that share an Iceberg table, an
+  accepted path or a state file never run together, and merge entities run one at a time. Reports and the summary
+  stay in config order; events of different entities interleave and observers are called from worker threads. On
+  the Polaris/SeaweedFS harness, 12 entities writing 12 Iceberg tables took 2.04 s with one worker and 1.03 s with
+  four. Without `strategy: parallel` nothing changes. Profiles that already set `strategy: parallel` for Dagster now
+  also run the entities of a `job_scope: domain` Job in parallel; peak memory then grows with the number of workers
+  (see `docs/profiles.md`).
+
+- **Creating a new Iceberg namespace no longer fails when two writers race (#1117).** Two entities, or two
+  processes such as concurrent Dagster Jobs, that both found a namespace missing and created it made the second
+  creation fail the run. A namespace that already exists by then now counts as created.
+
+- **A small JSON file is parsed once (#1115).** The key check before the read parsed every JSON file a second
+  time. A file of at most 8 MiB is now parsed once and its records reused; larger files keep the two passes, so
+  memory stays bounded. On 300 files of 200 records, JSON reading took 37-38% less CPU, and a 100,000-record file
+  about 28% less end to end.
+
 ## v0.8.7
 
 - **The Iceberg catalog is opened once per run (#1101).** Every catalog call used to start a new REST session

{floe_python-0.8.7 → floe_python-0.8.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.7
+Version: 0.8.8
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.7 → floe_python-0.8.8}/context/architecture.md

@@ -93,8 +93,11 @@ them (see `context/decisions/polars-and-arrow.md`).
    archive inputs (D32) → promote state claims → write run.json
 ```
 
-`floe run` runs entities in config order. A policy abort fails only its entity; any raised
-`FloeError` aborts the run (D36). Reports are written under `<report.path>/run_<id>/`, or
+`floe run` runs entities in config order, or on `execution.orchestration.max_concurrent_entities`
+threads when `strategy` is `parallel` (`run/parallel.py`: never two on one Iceberg table, accepted
+path or state file, merges one at a time; reports in config order). A policy abort fails only its
+entity; any raised `FloeError` aborts the run (D36): no new entity starts, running ones finish, the
+first error in config order is raised. Reports are written under `<report.path>/run_<id>/`, or
 `<report.path>/<domain>/run_<id>/` for a single-domain run (D37).
 
 ## Key contracts

{floe_python-0.8.7 → floe_python-0.8.8}/docs/profiles.md

@@ -31,6 +31,9 @@ metadata:                    # required
 execution:                   # optional
   runner:
     type: local              # required when execution is present
+  orchestration:             # optional; see "Running entities in parallel"
+    strategy: parallel       # optional; "sequential" (default) or "parallel"
+    max_concurrent_entities: 4  # optional; integer >= 1
 variables:                   # optional; flat string → string map
   KEY: value
 validation:                  # optional
@@ -355,6 +358,40 @@ Notes:
 - Full config validation (`floe validate -c config.yml --profile profiles/prod.yml`)
   additionally checks catalog/storage compatibility and entity sink bindings.
 
+## Running entities in parallel
+
+`execution.orchestration` also sets how many entities one `floe run` process runs at once
+(`floe run -c … -p …`, `floe run --manifest …` generated with the profile, and `floe.run(…, profile_path=…)`).
+Dagster reads the same fields to cap its own executor.
+
+```yaml
+execution:
+  runner: {type: local}
+  orchestration: {strategy: parallel, max_concurrent_entities: 4}
+```
+
+- **Opt-in.** Only `strategy: parallel` runs entities in parallel threads. `sequential`, no `orchestration`
+  block, or `max_concurrent_entities: 1` run them one after another in config order, as before.
+  `max_concurrent_entities` defaults to `min(4, CPU count)` when `strategy: parallel` leaves it unset.
+- **Never at the same time:** entities with the same Iceberg table (catalog, namespace, table), the same
+  accepted path or the same incremental state file; and `merge_scd1` / `merge_scd2` entities, which run
+  one at a time. They wait for each other while other entities proceed.
+- **Order.** `entity_started`, `file_*` and `entity_finished` events of different entities interleave, and
+  observers (including your `set_observer` callback) are called from worker threads. The
+  reports, the `run.summary.json` entity list and the console summary stay in config order.
+  Every entity's inputs are still listed before the first `entity_started`.
+- **Failure.** A policy `abort` fails only its entity. Any other error stops new entities from starting,
+  lets the running ones finish (they release or promote their incremental claims), then raises the first
+  error in config order. An interrupt (Ctrl-C) is handled the same way: the running entities finish first.
+- **Ephemeral disk.** Each running entity may hold up to its download prefetch window (8 files, about
+  512 MiB) in a temporary directory. Size ephemeral storage for `workers × 512 MiB` plus the largest file.
+- **Memory.** Each running entity holds its own chunk (up to 1M rows) in memory, and a merge entity's DuckDB
+  (`memory_limit` 4 GB) can run alongside non-merge entities. Peak memory grows with `max_concurrent_entities`:
+  size the pod for `workers` × one entity's working set (plus DuckDB when merges are configured), not for a sequential run.
+
+Existing profiles that already say `strategy: parallel` for Dagster now also run the entities of a
+`domain` Job in parallel inside its `floe run`.
+
 ## Hardening assessment and plan
 
 Current profile feature gaps and fixes:

{floe_python-0.8.7 → floe_python-0.8.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.7"
+version = "0.8.8"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.7 → floe_python-0.8.8}/scripts/test_config_schema.py

@@ -106,7 +106,7 @@ class ConfigSchemaTest(unittest.TestCase):
             types.SchemaEvolutionConfig: "schema_evolution", types.SchemaMismatchConfig: "schema_mismatch",
             types.ColumnConfig: "column",
         }  # `delta` and `duckdb` stay raw in the model (D1) and have no binding.
-        internal = {project.RootConfig: {"config_files"}, project.DomainConfig: {"resolved_incoming_dir"}, types.EntityConfig: {"id"}, types.SinkTarget: {"write_mode"}}
+        internal = {project.RootConfig: {"config_files", "entity_workers"}, project.DomainConfig: {"resolved_incoming_dir"}, types.EntityConfig: {"id"}, types.SinkTarget: {"write_mode"}}
         aliases = {types.SourceConfig: {"source_ref": "ref"}, types.ColumnConfig: {"column_type": "type"}}
         wire_extras = {
             project.RootConfig: {"sources", "include", "filesystems"},

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/__init__.py

@@ -136,8 +136,7 @@ def _profile(
         profile = parse_profile(path)
     except FloeError as e:
         raise FloeConfigError(f"failed to load profile {_debug(path)}: {e}") from e
-    return ProfileConfig(profile.name, profile.runner, profile.variables | (variables or {}),
-                         profile.catalogs, profile.storages, profile.lineage)  # fmt: skip
+    return replace(profile, variables=profile.variables | (variables or {}))
 
 
 def _load(config_path: str, profile: ProfileConfig | None = None) -> tuple[Path, _Root]:

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/_cli_output.py

@@ -1,6 +1,7 @@
 """`floe run` output: run events as log lines and the run summary (Rust: `crates/floe-cli/src/logging.rs`, `output.rs`)."""
 
 import json
+import threading
 import time
 from typing import Any
 
@@ -73,18 +74,20 @@ class Console:
     def __init__(self, log_format: str, run_id: str) -> None:
         self.log_format, self.run_id = log_format, run_id
         self.started = False
-        self.timings: list[dict[str, int]] = []  # per entity: input -> elapsed_ms
+        self.timings: dict[
+            str, dict[str, int]
+        ] = {}  # entity name -> input -> elapsed_ms
+        self._lock = threading.Lock()  # entities may run in parallel threads
 
     def __call__(self, event: dict[str, Any]) -> None:
         kind = event["event"]
-        self.started |= kind == "run_started"
-        if kind == "entity_started":
-            self.timings.append({})
-        elif kind == "file_finished" and self.timings:
-            self.timings[-1][event["input"]] = event["elapsed_ms"]
-        if self.log_format != "off":
-            fmt = event_json if self.log_format == "json" else event_text
-            print(fmt(event), flush=True)
+        with self._lock:
+            self.started |= kind == "run_started"
+            if kind == "file_finished":
+                self.timings.setdefault(event["entity"], {})[event["input"]] = event["elapsed_ms"]  # fmt: skip
+            if self.log_format != "off":
+                fmt = event_json if self.log_format == "json" else event_text
+                print(fmt(event), flush=True)
 
     def failed(self, err: FloeError, config: str | None) -> None:
         """Rust's `emit_failed_run_events`, after a `run_started` for `config` when the run never started.
@@ -154,7 +157,7 @@ def run_output(
     outcome: RunOutcome,
     entities: list[EntityConfig],
     mode: str,
-    timings: list[dict[str, int]],
+    timings: dict[str, dict[str, int]],
 ) -> str:
     """Rust's `format_run_output`; `mode` is `default`, `quiet` or `verbose`."""
     run_dir = outcome.run_dir  # a URI on remote storage: join with `/`, never `Path`
@@ -166,7 +169,17 @@ def run_output(
         results = dict.fromkeys(("files_total", "rows_total", "accepted_total", "rejected_total", "files_skipped"), 0)  # fmt: skip
         status, exit_code = "success", 0
     else:
-        blocks = [_entity(r, t, mode == "verbose") for r, t in zip(outcome.entity_reports, timings, strict=True)]  # fmt: skip
+        names = {
+            e.id: e.name for e in entities
+        }  # reports carry the id, `file_finished` events the name
+
+        def shown(report: dict[str, Any]) -> list[str]:
+            ident: str = report["entity"]["name"]
+            return _entity(
+                report, timings.get(names.get(ident, ident), {}), mode == "verbose"
+            )
+
+        blocks = [shown(r) for r in outcome.entity_reports]
         results, run = outcome.summary["results"], outcome.summary["run"]
         status, exit_code = run["status"], run["exit_code"]
     lines = ["DRY RUN MODE - No actual execution performed"] if outcome.dry_run is not None else []  # fmt: skip

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/checks/mismatch.py

@@ -119,19 +119,34 @@ def apply_mismatch(df: pl.DataFrame, plan: MismatchOutcome) -> pl.DataFrame:
     return df
 
 
-def json_input_columns(path: Path, mode: str = "array") -> list[str] | None:
-    """Sorted top-level keys of every record, read exactly as `read_json` reads them.
+def json_probe(
+    path: Path, mode: str, limit: int
+) -> tuple[list[str] | None, list[tuple[str, Any]] | None]:
+    """`json_input_columns` plus the file's parsed records when it has at most `limit` (else `None`).
 
-    D40: the union over all records, so a key only a later record carries is still an input
-    column (Rust 0.7 probed the first object only). `None` for a file without records (D41), which
-    differs from records that carry no keys (`[]`): those are still checked for missing columns.
+    A file that fits one chunk is then parsed once: the caller reads the records it got here.
     """
     if mode not in ("array", "ndjson"):
         raise FloeIoError(
             f"unsupported source.options.json_mode={mode} (allowed: array, ndjson)"
         )
-    # ponytail: the file is parsed twice (probe, then read); fold the probe into the read if it costs
     keys: set[str] | None = None
-    for _, record in json_records(path, mode):
-        keys = (keys or set[str]()) | {str(k) for k in record}
-    return None if keys is None else sorted(keys)
+    head: list[tuple[str, Any]] | None = []
+    for located in json_records(path, mode):
+        keys = keys or set[str]()
+        keys.update(str(k) for k in located[1])
+        if head is not None and len(head) < limit:
+            head.append(located)
+        else:  # a longer file is read again, chunk by chunk, so it keeps nothing
+            head = None
+    return (None if keys is None else sorted(keys)), head
+
+
+def json_input_columns(path: Path, mode: str = "array") -> list[str] | None:
+    """Sorted top-level keys of every record, read exactly as `read_json` reads them.
+
+    D40: the union over all records, so a key only a later record carries is still an input
+    column (Rust 0.7 probed the first object only). `None` for a file without records (D41), which
+    differs from records that carry no keys (`[]`): those are still checked for missing columns.
+    """
+    return json_probe(path, mode, 0)[0]

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/config/profile.py

@@ -1,7 +1,7 @@
 """Environment profiles, ported from `profile/parse.rs` and `profile/validate.rs`."""
 
 import os
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
 from dataclasses import dataclass
 from pathlib import Path
 from typing import cast
@@ -395,12 +395,21 @@ def validate_profile(profile: ProfileConfig) -> None:
         check_lineage(profile.lineage, "profile.lineage")
 
 
+def entity_workers(orchestration: Mapping[str, object] | None) -> int:
+    """How many entities a run may run at once: 1 unless `strategy` is `parallel`, then `max_concurrent_entities` or `min(4, cpus)`."""
+    if not orchestration or orchestration.get("strategy") != "parallel":
+        return 1
+    limit = orchestration.get("max_concurrent_entities")
+    return limit if isinstance(limit, int) else min(4, os.cpu_count() or 1)
+
+
 def apply_profile(config: RootConfig, profile: ProfileConfig) -> RootConfig:
     """The profile's blocks replace the config's, as `apply_profile_catalogs`, `_storages` and `_lineage` do."""
     blocks = {
         "catalogs": profile.catalogs,
         "storages": profile.storages,
         "lineage": profile.lineage,
+        "entity_workers": entity_workers(profile.orchestration),
     }
     return config.model_copy(update={k: v for k, v in blocks.items() if v is not None})
 

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/config/project.py

@@ -93,6 +93,7 @@ class RootConfig(BaseModel):
     lineage: LineageConfig | None
     entities: list[EntityConfig]
     config_files: list[Path]
+    entity_workers: int = 1  # profile/manifest orchestration; not a config key
 
 
 @dataclass

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/iceberg/catalog.py

@@ -7,11 +7,12 @@ import tempfile
 import threading
 import weakref
 from collections.abc import Generator, Sequence
-from contextlib import contextmanager
+from contextlib import contextmanager, suppress
 from urllib.parse import urlsplit
 
 import duckdb
 from pyiceberg.catalog import Catalog, load_catalog
+from pyiceberg.exceptions import NamespaceAlreadyExistsError
 from pyiceberg.partitioning import PartitionField, PartitionSpec
 from pyiceberg.schema import Schema
 from pyiceberg.table import Table
@@ -350,7 +351,10 @@ def ensure_table(
     if not namespace_exists:
         if not create_namespace:
             raise FloeRunError(f"entity.name={entity} iceberg namespace {namespace} does not exist and create_database_if_missing is false")  # fmt: skip
-        with run_error("rest catalog namespace create failed"):
+        with (
+            run_error("rest catalog namespace create failed"),
+            suppress(NamespaceAlreadyExistsError),
+        ):  # another entity or process created it first
             catalog.create_namespace(ident)
     identifier = (namespace, name)
     with run_error("rest catalog table_exists check failed"):

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/lineage/observer.py

@@ -110,7 +110,6 @@ def _warn(message: str) -> None:
 class OpenLineageObserver:
     """A run observer (`floe.run.events.Observer`) posting OpenLineage `RunEvent`s."""
 
-    # ponytail: unsynchronised breaker counters; add a lock when entities run in parallel.
     def __init__(
         self, config: LineageConfig, entities: Sequence[EntityConfig], config_path: str
     ) -> None:
@@ -127,12 +126,14 @@ class OpenLineageObserver:
         self.run_uuid: str | None = None
         self.consecutive_failures = 0
         self.circuit_open = False
+        self._lock = threading.Lock()  # entities may run in parallel threads
 
     def __call__(self, event: dict[str, Any]) -> None:
         kind, ts = event["event"], event["ts_ms"]
         job = self.events.run_job_name
         if kind == "run_started":
-            self.consecutive_failures, self.circuit_open = 0, False
+            with self._lock:
+                self.consecutive_failures, self.circuit_open = 0, False
             self.run_uuid = str(uuid.uuid4())
             self._post("START", ts, self.run_uuid, job)
         elif kind == "entity_started":
@@ -171,19 +172,19 @@ class OpenLineageObserver:
             time.sleep(delay)
             retryable = self._attempt(data)
             if retryable is None:
-                self.consecutive_failures = 0
+                with self._lock:
+                    self.consecutive_failures = 0
                 return
             if not retryable:
                 break
         _warn(f"POST {self.url} failed — event dropped")
-        self.consecutive_failures += 1
-        failures = self.consecutive_failures
-        if (
-            failures
-            >= (3 if self.config.max_failures is None else self.config.max_failures)
-            and not self.circuit_open
-        ):
-            self.circuit_open = True
+        with self._lock:
+            self.consecutive_failures += 1
+            failures = self.consecutive_failures
+            limit = 3 if self.config.max_failures is None else self.config.max_failures
+            opening = failures >= limit and not self.circuit_open
+            self.circuit_open |= opening
+        if opening:
             _warn(f"disabled for this run after {failures} consecutive failures — check endpoint {self.url}")  # fmt: skip
 
     def _attempt(self, data: bytes) -> bool | None:
@@ -251,6 +252,7 @@ def run_with_lineage(
         return run(
             path, root, run_id=run_id, entities=entities, dry_run=dry_run,
             full_refresh=full_refresh, observer=observe, config_uri=config_uri,
+            workers=root.entity_workers,
         )  # fmt: skip
     except FloeError:
         if lineage:

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/manifest_run.py

@@ -12,6 +12,7 @@ from typing import Any, TypeVar, cast, get_args
 
 from pydantic import BaseModel, ValidationError
 
+from floe.config.profile import entity_workers
 from floe.config.project import (
     LineageConfig,
     ProjectMetadata,
@@ -265,6 +266,9 @@ def load_manifest(location: str | Path) -> tuple[RootConfig, list[EntityConfig],
         spec = m["spec_version"]
         entities = [_entity(e, _named_local(m)) for e in m["entities"]]
         report_base = m["report_base_uri"]
+        workers = entity_workers(
+            cast(Json, m.get("execution") or {}).get("orchestration")
+        )
         meta = m.get("metadata")  # D20: dumped without its None fields
         metadata = None if meta is None else ProjectMetadata.model_validate(_omitted_as_none(ProjectMetadata, meta))  # fmt: skip
     except (ValueError, KeyError, TypeError, AttributeError) as e:
@@ -289,6 +293,7 @@ def load_manifest(location: str | Path) -> tuple[RootConfig, list[EntityConfig],
         lineage=_section(LineageConfig, m, "lineage"),
         entities=entities,
         config_files=[],
+        entity_workers=workers,
     )
     observed = [_lineage_entity(e, me) for e, me in zip(entities, m["entities"])]
     return root, observed, _replay_base(location, m)

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/read/json.py

@@ -15,6 +15,11 @@ from floe.config.types import ColumnConfig, SourceOptions
 from floe.errors import FloeIoError
 
 Selector = list[str | int]  # a field name or an array index
+JSON_CHUNK_ROWS = 1_000_000
+# ponytail: a file of at most this many bytes is parsed once, its dicts held until read (about 16x its size
+# in memory, so 8 MiB costs about 130 MiB); larger files parse twice (key pass, then chunks).
+# Upgrade path: build the first chunk's rows while probing.
+JSON_SINGLE_PASS_BYTES = 8 * 1024 * 1024
 
 
 @dataclass(frozen=True)
@@ -188,13 +193,15 @@ def read_json(
     options: SourceOptions,
     columns: Sequence[ColumnConfig],
     cast_mode: str = "strict",
-    chunk_rows: int = 1_000_000,
+    chunk_rows: int = JSON_CHUNK_ROWS,
+    records: Sequence[tuple[str, Any]] | None = None,
 ) -> Iterator[JsonChunk]:
     """Yield the rows in chunks of at most `chunk_rows`, columns named by selector; always at least one.
 
     `options.json_mode` is `array` (default) or `ndjson`. A selector that finds nothing is null.
     Objects and arrays are an error unless `cast_mode` is `coerce`, which keeps their compact JSON.
     Scalars are stringified with Python's `str`; Polars would render `1.0` as `1`, so it is not used.
+    `records` are the file's already parsed `json_records`, read instead of parsing the file again.
     """
     mode = options.json_mode or "array"
     if mode not in ("array", "ndjson"):
@@ -221,7 +228,7 @@ def read_json(
     exact = [decimal_type(c.column_type) is not None for c in columns]
     rows: list[list[str | None]] = []
     done = 0
-    for where, value in json_records(path, mode):
+    for where, value in json_records(path, mode) if records is None else records:
         rows.append(
             [
                 _cell(value, p, n, cast_mode, where, e)

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/run/__init__.py

@@ -1,4 +1,4 @@
-"""A run: entity selection, run context, sequential or parallel entities, dry run, full refresh, events (Rust: `run/mod.rs`, `run/context.rs`)."""
+"""A run: entity selection, run context, entities in config order or in parallel threads (`workers` > 1), dry run, full refresh, events (Rust: `run/mod.rs`, `run/context.rs`)."""
 
 import time
 from collections.abc import Sequence
@@ -23,6 +23,7 @@ from floe.report.summary import (
 )
 from floe.run.entity import EntityRun, list_source, resolve_source, run_entity
 from floe.run.events import Emitter, Observer
+from floe.run.parallel import exclusive_keys, run_parallel
 from floe.storage.inputs import ResolvedInputs
 from floe.storage.paths import normalize_local_path
 from floe.storage.resolve import clear_storage_clients, resolve_path, storage_client
@@ -103,8 +104,12 @@ def run(
     full_refresh: bool = False,
     observer: Observer | None = None,
     config_uri: str | None = None,
+    workers: int = 1,
 ) -> RunOutcome:
-    """Run the selected entities in config order, writing each entity's report as it finishes. A failing entity raises: no summary, as Rust."""
+    """Run the selected entities, writing each entity's report as it finishes. A failing entity raises: no summary, as Rust.
+
+    One worker runs them in config order. More run them in parallel threads (observers are called from those threads), never two on the same target; reports and summary stay in config order.
+    """
     clear_storage_clients()
     started, timer = _rfc3339(), time.monotonic()
     clear_catalogs()
@@ -159,14 +164,21 @@ def run(
         archive = full_refresh and e.sink.archive and e.incremental_mode in ("none", "archive")  # fmt: skip
         if not archive:
             listed[e.id] = inputs(e)
-    reports: list[tuple[str, dict[str, Any]]] = []
     client = storage_client(report_storage)
-    # D36: a policy abort fails only its entity; a FloeError still stops the run
-    for e in selected:
+
+    def finish(e: EntityConfig) -> dict[str, Any]:
         report = one(e).report
-        reports.append((id_path(e), report))
         write_entity_report(run_path, id_path(e), report, client)
         _finished(emit, e.id, report)
+        return report
+
+    # D36: a policy abort fails only its entity; a FloeError still stops the run
+    if workers > 1 and len(selected) > 1:
+        keys = [exclusive_keys(e, config_dir, root.storages, root.catalogs) for e in selected]  # fmt: skip
+        done = run_parallel(selected, keys, workers, finish)
+    else:
+        done = [finish(e) for e in selected]
+    reports = [(id_path(e), r) for e, r in zip(selected, done, strict=True)]
     summary = build_summary(
         root.version, reports,
         run_id=run_id, started_at=started, finished_at=_rfc3339(),

{floe_python-0.8.7 → floe_python-0.8.8}/src/floe/run/entity.py

@@ -21,7 +21,7 @@ from floe.checks.mismatch import (
     MismatchOutcome,
     apply_mismatch,
     declared_names,
-    json_input_columns,
+    json_probe,
     plan_mismatch,
 )
 from floe.checks.normalize import (
@@ -59,7 +59,7 @@ from floe.iceberg.write import find_committed
 from floe.pii import apply_pii_masking
 from floe.read.avro import read_avro
 from floe.read.csv import RAGGED_MESSAGE, read_csv
-from floe.read.json import read_json
+from floe.read.json import JSON_CHUNK_ROWS, JSON_SINGLE_PASS_BYTES, read_json
 from floe.read.orc import read_orc
 from floe.read.parquet import read_parquet
 from floe.read.xlsx import read_xlsx
@@ -118,7 +118,7 @@ class EntityRun:
     aborted: bool
 
 
-def _chunks(entity: EntityConfig, path: Path, names: list[str]) -> Chunks:
+def _chunks(entity: EntityConfig, path: Path, names: list[str], head: list[tuple[str, Any]] | None = None) -> Chunks:  # fmt: skip
     src = entity.source
     match src.format:
         case "csv" | "tsv":
@@ -139,7 +139,7 @@ def _chunks(entity: EntityConfig, path: Path, names: list[str]) -> Chunks:
             for x in read_xml(path, src.options, entity.schema_.columns[: len(names)]):
                 yield x.offset, x.df, None
         case "json":  # #998
-            for j in read_json(path, src.options, entity.schema_.columns[: len(names)], src.cast_mode):  # fmt: skip
+            for j in read_json(path, src.options, entity.schema_.columns[: len(names)], src.cast_mode, JSON_CHUNK_ROWS, head):  # fmt: skip
                 yield j.offset, j.df, None
         case other:
             raise FloeRunError(
@@ -411,7 +411,9 @@ def _run_file(plan: _Plan, file: InputFile, path: Path, spill: Path) -> tuple[di
             first, probe = [], xml_columns(path, entity.source.options)
         elif entity.source.format == "json":
             # D40: every record's top-level keys; D41: only a file without records has nothing to compare
-            json_keys = json_input_columns(path, entity.source.options.json_mode or "array")  # fmt: skip
+            # a file listed as small is parsed once: the read reuses the probe's records (limit 0 keeps none)
+            json_keys, head = json_probe(path, entity.source.options.json_mode or "array", JSON_CHUNK_ROWS if file.source_size is not None and file.source_size <= JSON_SINGLE_PASS_BYTES else 0)  # fmt: skip
+            chunks = _chunks(entity, path, file_names, head)
             first, probe = [], declared if json_keys is None else json_keys
         else:  # the first chunk's columns are the input columns the plan compares
             first = [next(chunks)]

floe_python-0.8.8/src/floe/run/parallel.py

@@ -0,0 +1,71 @@
+"""Entities in parallel threads (#1105): the schedule keeps entities that share a resource apart and the failure rule of D36."""
+
+from collections.abc import Callable, Sequence
+from concurrent.futures import Future, ThreadPoolExecutor, wait
+from pathlib import Path
+from typing import Any
+
+from floe.config.storage import CatalogsConfig, StoragesConfig
+from floe.config.types import EntityConfig
+from floe.errors import FloeError
+from floe.iceberg.sink import resolve_target
+from floe.run.entity import state_uri
+from floe.storage.resolve import resolve_path
+
+MERGE = "merge"  # every merge entity shares this key: DuckDB memory, and Polaris conflicts on one table
+
+
+def _canonical(uri: str) -> str:
+    """One spelling per destination: a local path resolved through symlinks, a remote URI without a trailing `/`."""
+    if uri.startswith("local://"):
+        return f"local://{Path(uri.removeprefix('local://')).resolve()}"
+    return uri.rstrip("/")
+
+
+def exclusive_keys(e: EntityConfig, config_dir: Path, storages: StoragesConfig | None, catalogs: CatalogsConfig | None) -> frozenset[str]:  # fmt: skip
+    """What `e` writes that no other running entity may touch: its accepted path, Iceberg table, state file, and the merge slot."""
+    sink = e.sink
+    try:
+        keys = {_canonical(resolve_path(storages, config_dir, sink.accepted.storage, sink.accepted.path, f"entity.name={e.name} sink.accepted.storage")[0])}  # fmt: skip
+        if sink.accepted.format == "iceberg":
+            t = resolve_target(storages, catalogs, config_dir, e)
+            keys.add(f"iceberg://{t.catalog.name}/{t.namespace}/{t.table}")
+        if e.incremental_mode == "file":
+            keys.add(_canonical(state_uri(e, config_dir, storages)[0]))
+    except FloeError:
+        return frozenset()  # `run_entity` raises the same error when it runs
+    if sink.write_mode.startswith("merge"):
+        keys.add(MERGE)
+    return frozenset(keys)
+
+
+def run_parallel(entities: Sequence[EntityConfig], keys: Sequence[frozenset[str]], workers: int, task: Callable[[EntityConfig], dict[str, Any]]) -> list[dict[str, Any]]:  # fmt: skip
+    """Run `task` over `entities` on `workers` threads, never two with a key in common; the reports in `entities` order.
+
+    After a failure no entity starts, the running ones finish (and release their claims), and the first error in `entities` order is raised.
+    """
+    reports: dict[int, dict[str, Any]] = {}
+    errors: dict[int, BaseException] = {}
+    pending = list(range(len(entities)))
+    running: dict[Future[dict[str, Any]], int] = {}
+    busy: set[str] = set()
+    with ThreadPoolExecutor(workers, thread_name_prefix="floe-entity") as pool:
+        while pending or running:
+            for i in list(pending):
+                if len(running) == workers:
+                    break
+                if not keys[i] & busy:
+                    pending.remove(i)
+                    busy |= keys[i]
+                    running[pool.submit(task, entities[i])] = i
+            for future in wait(running, return_when="FIRST_COMPLETED").done:
+                i = running.pop(future)
+                busy -= keys[i]
+                if error := future.exception():
+                    errors[i] = error
+                    pending.clear()
+                else:
+                    reports[i] = future.result()
+    if errors:
+        raise errors[min(errors)]
+    return [reports[i] for i in range(len(entities))]