floe-python 0.8.6__tar.gz → 0.8.7__tar.gz

{floe_python-0.8.6 → floe_python-0.8.7}/CHANGELOG.md

@@ -2,6 +2,34 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.7
+
+- **The Iceberg catalog is opened once per run (#1101).** Every catalog call used to start a new REST session
+  (a config request, plus an OAuth token request when a credential is set), and an `append` entity did it twice.
+  A run now reuses one catalog per catalog definition and storage, so N entities no longer pay up to 2N
+  initialisations. The catalog is opened again at the start of each run.
+
+- **S3 clients are reused within a run (#1100).** Each `S3Client` builds its own boto3 session, about 55-120 ms,
+  and one S3 entity built 6-8 of them (source, state, accepted, rejected, archive, report). A run now builds one
+  client per storage definition: a two-entity run on one storage went from 5 clients to 1. `AWS_*` variables and
+  credentials are read again at the start of each run, and by `inspect_entity_state()` and
+  `reset_entity_state()`.
+
+- **Remote inputs are downloaded ahead of the file being read (#1102).** Each pending S3 file used to be
+  downloaded only when its turn came, so the read loop waited on every GET. An entity now downloads up to 8
+  files, and about 512 MiB, ahead of the one it is validating (a single larger file still downloads, alone).
+  Files are still processed in the same order, a failed download still raises its own error when that file's turn
+  comes, and local sources are unchanged. With a simulated 0.1 s download per file, 16 files went from 1.72 s to
+  0.24 s.
+
+- **Small CSV files cost fewer Polars plans (#1104).** A clean 200-row file ran 16 query plans between the
+  reader and the writer and now runs 6, with identical output. On 1,200 such files (8 entities of 150 files,
+  Parquet sink on local disk) the run was about 26-28% faster, 7.45 s to 5.38 s in the last measurement.
+
+- **`poll_interval_seconds` and `job_scope` are documented (#1103).** `docs/profiles.md` and
+  `context/orchestrators.md` describe the Kubernetes runner's status poll (default 10 s, a Job that finishes
+  early is noticed at the next poll) and when to choose `job_scope: domain` over `entity`.
+
 ## v0.8.6
 
 - **Merge modes keep a key's latest version when it repeats across pending files (#1090).** With `merge_scd1`
@@ -63,18 +91,19 @@ All notable changes to Floe are documented in this file.
 
 ## Unreleased
 
-### dagster-floe
-
-- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
-  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
-  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
-  `merge_scd1` / `merge_scd2` entity needs a Floe version that accepts `--full-refresh` with merge modes.
-
 ### airflow-floe
 
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
   dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
+## dagster-floe v0.4.3
+
+- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
+  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
+  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
+  `merge_scd1` entity needs Floe 0.8.6 or newer, which rebuilds the table from the inputs; `merge_scd2`
+  still rejects `--full-refresh`.
+
 ## dagster-floe v0.4.2
 
 - **Silver lineage events from a `kubernetes_job` run now carry the Dagster parent run (#1076).** The Job

{floe_python-0.8.6 → floe_python-0.8.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.6
+Version: 0.8.7
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.6 → floe_python-0.8.7}/context/orchestrators.md

@@ -211,6 +211,7 @@ When `type = kubernetes_job`, the manifest runner definition supports:
 | `node_selector` | object | Pod `nodeSelector` |
 | `tolerations` | array | Pod `tolerations`, verbatim |
 | `backoff_limit` | integer >= 0 | Job `backoffLimit` (default `0`) |
+| `poll_interval_seconds` | integer >= 1 | Seconds between status reads of the Job (default `10`) |
 | `env_from` | array | Container `envFrom`, verbatim, e.g. `[{secretRef: {name: floe-env}}]` |
 
 These fields are populated automatically when `floe manifest generate` is given a profile with a `kubernetes_job` runner block. The `dagster-floe[kubernetes]` optional extra expresses the Kubernetes client requirement.
@@ -220,6 +221,8 @@ These fields are populated automatically when `floe manifest generate` is given
 - **`entity`** (default, also when unset): one Job per entity, named `floe-<entity>-<id>`. Pod start is paid once per entity; each entity has its own timeout, TTL and failure. Use it for a few large entities, or when entities need isolation.
 - **`domain`**: entities of one domain whose runner has `job_scope: domain` form one subsettable multi-asset per (domain, runner), so an entity with another runner gets its own Job. Materializing it launches **one** Job named `floe-<domain>-<id>` running `floe run --manifest <uri> --entities a,b,c` (`--domain <d>` when they are every entity of the domain in the manifest) for the selected entities in manifest order, one after another, so Polaris never sees overlapping commits from the Job. The Job is sized from `domain_resources[<domain>]`, else `resources`; `timeout_seconds`, TTL and polling apply to the one Job. Use it when pod start dominates (many small entities). Select the accepted assets: a rejected asset is materialized only when it is selected too, and the asset job selects both. The `floe/table` op tag (#924) is not set on a domain asset.
 
+**Choosing a scope and a poll interval.** The runner reads the Job's status every `poll_interval_seconds`, so a Job that finishes early is only noticed at the next poll: with the default 10 s, a 3 s Job is seen about 7 s late, a flat per-Job cost. `entity` pays pod start and that polling delay once per entity, but entities are isolated and run in parallel through Dagster's executor. `domain` pays both once per domain, but runs its entities one after another. For short Jobs, lower `poll_interval_seconds` (for example `2`), at the cost of more Kubernetes API reads.
+
 Dagster still reports per entity (materialization, rows, rejected counts, report URI, asset checks) from the run's `run_finished` event and summary.
 
 **After a failure** each entity is reported from its own status in the run summary: `rejected` and `success_with_warnings` entities are materialized, an `aborted` / `failed` entity fails, and an entity absent from the summary never ran and is reported as *not run* (listed in the step failure's `not_run_entities`), not as its own failure. Since D36 (#959) a policy `severity: abort` fails only its entity and the run continues with the next one. A global failure (access, memory, any raised `FloeError`) still aborts the run. Re-executing the step reruns the selected entities; subset it to rerun only some. Report URIs come from `run_finished` (`summary_uri`, `entity_report_uris`), never built by hand (#960).

{floe_python-0.8.6 → floe_python-0.8.7}/docs/profiles.md

@@ -238,12 +238,22 @@ execution:
     node_selector: {pool: batch}
     tolerations: [{key: batch, operator: Exists, effect: NoSchedule}]   # verbatim
     backoff_limit: 2                         # Job backoffLimit, default 0
+    poll_interval_seconds: 2                 # seconds between Job status reads, default 10
     env_from: [{secretRef: {name: floe-env}}]   # container envFrom, verbatim
 ```
 
 `labels`, `annotations` and `node_selector` are string maps, `backoff_limit` is a non-negative integer,
 and `security_context`, `tolerations` and `env_from` are copied as written: Kubernetes validates them.
 
+`poll_interval_seconds` is an integer of at least 1 (default 10): dagster-floe reads the Job's status that
+often, so a Job that finishes early is only noticed at the next poll. A lower value such as `2` shortens
+short Jobs at the cost of more Kubernetes API reads.
+
+`job_scope` (`entity` by default, or `domain`) sets how many Jobs a run launches. `entity` starts one Job
+per entity: each pays pod start and one poll interval, but they are isolated and run in parallel through
+Dagster's executor. `domain` starts one Job per domain: pod start and polling are paid once, but its
+entities run one after another.
+
 ---
 
 ## Using profile variables in a Floe config

{floe_python-0.8.6 → floe_python-0.8.7}/orchestrators/dagster-floe/README.md

@@ -42,6 +42,7 @@ pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.3,<0.9 for t
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.3 | 0.8.3 or newer (the `floe/full_refresh` run tag on a `merge_scd1` entity needs 0.8.6) | `0.8` |
 | 0.4.2 | 0.8.3 or newer | `0.8` |
 | 0.4.1 | 0.8.3 or newer (`kubernetes_job` runs pass the profile as `-p env:FLOE_PROFILE_CONTENT`) | `0.8` |
 | 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |

{floe_python-0.8.6 → floe_python-0.8.7}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.4.2"
+version = "0.4.3"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"

{floe_python-0.8.6 → floe_python-0.8.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.6"
+version = "0.8.7"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/__init__.py

@@ -46,6 +46,7 @@ from floe.run.events import (
 )
 from floe.state import dump_state, read_state, reset
 from floe.storage.paths import normalize_local_path
+from floe.storage.resolve import clear_storage_clients
 from floe.storage.types import StorageClient
 
 _observer: Callable[[RunEvent], None] | None = None
@@ -281,6 +282,7 @@ def extract_config_env_vars(config_path: str) -> dict[str, str]:
 
 
 def _state(config_path: str, entity_name: str) -> tuple[str, str, str, StorageClient]:
+    clear_storage_clients()
     path, root = _load(config_path)
     [entity] = _run.select_entities(root, [entity_name])
     uri, client = state_uri(entity, normalize_local_path(path.parent), root.storages)

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/checks/normalize.py

@@ -115,4 +115,6 @@ def normalize_columns(df: pl.DataFrame, strategy: str | None) -> pl.DataFrame:
                 f"normalized input column collision: {seen[new]} and {old} -> {new}"
             )
         seen[new] = old
-    return df.rename(names)
+    renamed = df.clone()
+    renamed.columns = list(names.values())  # a rename is a query plan, this is not
+    return renamed

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/checks/rows.py

@@ -109,6 +109,10 @@ def split_rows(
     sources: Mapping[str, str] | None = None,
 ) -> tuple[pl.DataFrame, pl.DataFrame]:
     """`(accepted, rejected)`; rejected adds `__floe_row_index` (UInt64, row position in `df`) and `__floe_errors`."""
+    # a null mask passes the row, as in `error_column`
+    if not any(c.mask.any() for c in checks):
+        none = [pl.Series(ROW_INDEX, [], pl.UInt64), pl.Series(ERRORS, [], pl.String)]
+        return df, df.clear().with_columns(none)  # replaces a clashing column, as below
     errors = error_column(df.height, checks, formatter, sources)
     rejected = df.with_columns(
         pl.int_range(pl.len(), dtype=pl.UInt64).alias(ROW_INDEX), errors

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/iceberg/catalog.py

@@ -4,6 +4,7 @@ import os
 import re
 import shutil
 import tempfile
+import threading
 import weakref
 from collections.abc import Generator, Sequence
 from contextlib import contextmanager
@@ -83,8 +84,34 @@ def _credential_props(catalog: str, credential: str) -> dict[str, str]:
     return {"credential": credential.removeprefix("client_credentials:")}
 
 
+_CATALOGS: dict[tuple[CatalogDefinition, StorageDefinition | None], Catalog] = {}
+_CATALOGS_LOCK = threading.Lock()
+
+
+def clear_catalogs() -> None:
+    """Called when a run starts: a catalog is reused within a run only, never across runs (#1101)."""
+    with _CATALOGS_LOCK:
+        _CATALOGS.clear()
+
+
 def open_catalog(
     definition: CatalogDefinition, storage: StorageDefinition | None = None
+) -> Catalog:
+    """One catalog per definition and storage until `clear_catalogs`, so the REST init (config, OAuth token) happens once.
+
+    Safe to call from several threads (one init per key). PyIceberg 0.12 refetches an expired OAuth token
+    itself: its REST calls retry once on a 401 after a new client-credentials request; a static `token:` has
+    nothing to refetch, as with a freshly opened catalog. A failed init is not kept.
+    """
+    key = (definition, storage)
+    with _CATALOGS_LOCK:
+        if key not in _CATALOGS:
+            _CATALOGS[key] = _open_catalog(definition, storage)
+        return _CATALOGS[key]
+
+
+def _open_catalog(
+    definition: CatalogDefinition, storage: StorageDefinition | None
 ) -> Catalog:
     """Credentials for S3 and Glue come from the environment; the catalog never vends them (as in Rust)."""
     props = {

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/read/csv.py

@@ -79,15 +79,16 @@ def _scan_args(options: SourceOptions) -> _Args:
 
 def _columns(path: Path, args: _Args, declared: Sequence[str]) -> list[str]:
     try:
-        probe = args.scan(path, n_rows=0 if args.has_header else 1).collect()
+        probe = args.scan(path, n_rows=0 if args.has_header else 1)
+        if args.has_header:  # the schema is the header: no plan to run
+            return probe.collect_schema().names()
+        width = probe.collect().width
     except (pl.exceptions.PolarsError, OSError) as err:
         # Rust's polars prints the error kind: `no data: empty CSV`
         kind = "no data: " if isinstance(err, pl.exceptions.NoDataError) else ""
         raise FloeIoError(f"csv header read failed: {kind}{err}") from err
-    if args.has_header:
-        return probe.columns
-    names = list(declared[: probe.width])
-    return names + [f"extra_column_{i + 1}" for i in range(len(declared), probe.width)]
+    names = list(declared[:width])
+    return names + [f"extra_column_{i + 1}" for i in range(len(declared), width)]
 
 
 def _field_counts(path: Path, args: _Args, skip: int) -> Iterator[int]:

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/run/__init__.py

@@ -11,6 +11,7 @@ from floe.checks.rows import Formatter
 from floe.config.project import RootConfig
 from floe.config.types import EntityConfig
 from floe.errors import FloeConfigError, FloeError
+from floe.iceberg.catalog import clear_catalogs
 from floe.report.summary import (
     build_summary,
     id_path,
@@ -24,7 +25,7 @@ from floe.run.entity import EntityRun, list_source, resolve_source, run_entity
 from floe.run.events import Emitter, Observer
 from floe.storage.inputs import ResolvedInputs
 from floe.storage.paths import normalize_local_path
-from floe.storage.resolve import resolve_path, storage_client
+from floe.storage.resolve import clear_storage_clients, resolve_path, storage_client
 
 
 @dataclass(frozen=True)
@@ -104,7 +105,9 @@ def run(
     config_uri: str | None = None,
 ) -> RunOutcome:
     """Run the selected entities in config order, writing each entity's report as it finishes. A failing entity raises: no summary, as Rust."""
+    clear_storage_clients()
     started, timer = _rfc3339(), time.monotonic()
+    clear_catalogs()
     selected = select_entities(root, entities)
     if full_refresh:
         selected = [_refresh(e) for e in selected]

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/run/entity.py

@@ -5,7 +5,7 @@ import tempfile
 import threading
 import time
 from collections.abc import Callable, Generator, Iterable, Iterator
-from contextlib import suppress
+from contextlib import closing, suppress
 from dataclasses import dataclass, field, replace
 from datetime import datetime
 from functools import partial
@@ -85,7 +85,6 @@ from floe.storage.inputs import (
     InputFile,
     NoInputsError,
     ResolvedInputs,
-    localize_input,
     resolve_inputs,
     suffixes_for_format,
 )
@@ -99,6 +98,7 @@ from floe.storage.partitioning import (
     window,
 )
 from floe.storage.paths import normalize_local_path
+from floe.storage.prefetch import localized_inputs
 from floe.storage.resolve import resolve_path, storage_client
 from floe.storage.s3 import S3Client
 from floe.storage.types import StorageClient
@@ -335,27 +335,31 @@ def _typed(
 ) -> tuple[pl.DataFrame, list[RowCheck]]:
     entity, src = plan.entity, plan.entity.source
     # D28 (#913): the derived columns join after the file's own columns are normalised and matched
-    df = apply_mismatch(normalize_columns(raw, plan.strategy), mismatch).with_columns(
-        derived
-    )
-    df = df.select(
-        pl.col(n) if n in df.columns else pl.lit(None, pl.String).alias(n)
-        for n in plan.names
-    )
+    df = apply_mismatch(normalize_columns(raw, plan.strategy), mismatch)
+    if derived:
+        df = df.with_columns(derived)
+    if df.columns != plan.names:
+        df = df.select(
+            pl.col(n) if n in df.columns else pl.lit(None, pl.String).alias(n)
+            for n in plan.names
+        )
     columns = entity.schema_.columns
+    tag = "_e_cast_"  # the mask columns' prefix: no declared column may start with it
+    while any(n.startswith(tag) for n in plan.names):
+        tag = f"_{tag}"
     try:
         casts = [
             cast_column(n, c.column_type, src.cast_mode, df.schema[n])
             for n, c in zip(plan.names, columns, strict=True)
         ]
-        typed = df.with_columns(t for t, _ in casts)
-        masks = df.with_columns(
-            m for _, m in casts
-        )  # unlike select, broadcasts constant masks
+        # one plan for both; unlike select, with_columns broadcasts the constant masks
+        masks = [m.alias(f"{tag}{n}") for n, (_, m) in zip(plan.names, casts, strict=True)]  # fmt: skip
+        both = df.with_columns([t for t, _ in casts] + masks)
+        typed = both[plan.names]
     except pl.exceptions.InvalidOperationError as err:
         raise FloeRunError(f"entity.name={entity.name} cast failed: {err}") from err
     cast_checks = [
-        RowCheck("cast_error", n, CAST_ERROR_MESSAGE, masks[f"_e_cast_{n}"])
+        RowCheck("cast_error", n, CAST_ERROR_MESSAGE, both[f"{tag}{n}"])
         for n, c in zip(plan.names, columns, strict=True)
         if c.column_type != "string"
     ]  # fmt: skip
@@ -940,14 +944,18 @@ def _run_entity(
     def accepted_frames() -> Generator[pl.DataFrame]:
         """Files are read as the sink pulls (#1056); `files` and `complete` are final once it is exhausted."""
         try:
-            # Rust `localize_input`: a remote input is read from a local copy
-            with tempfile.TemporaryDirectory(prefix="floe-") as temp_dir:
+            queue = pending[::-1] if latest_first else pending
+            # Rust `localize_input`: a remote input is read from a local copy; #1102: downloaded ahead, and closed before the temp dir goes
+            with (
+                tempfile.TemporaryDirectory(prefix="floe-") as temp_dir,
+                closing(localized_inputs(queue, downloader, Path(temp_dir))) as inputs,
+            ):
                 if pending and sink.write_mode == "append" and plan.tracker.keys and (target is None or plan.table is not None):  # fmt: skip
                     plan.history = _key_cache(plan, accepted_client, base, Path(temp_dir))  # fmt: skip
-                for f in pending[::-1] if latest_first else pending:
+                for f in queue:
                     emit("file_started", entity=entity.name, input=f.source_uri)
                     timer = time.monotonic()
-                    local = localize_input(f, downloader, Path(temp_dir))
+                    local = next(inputs)
                     report, kept = _run_file(plan, f, local.local_path, Path(temp_dir))
                     if local.is_ephemeral:
                         local.local_path.unlink()

floe_python-0.8.7/src/floe/storage/prefetch.py

@@ -0,0 +1,46 @@
+"""Download the inputs of one entity a few files ahead of the one being read (#1102)."""
+
+from collections import deque
+from collections.abc import Generator, Iterable
+from concurrent.futures import Future, ThreadPoolExecutor
+from pathlib import Path
+
+from floe.storage.inputs import InputFile, LocalInputFile, localize_input
+from floe.storage.types import Downloader
+
+# ponytail: a fixed window of 8 files and 512 MiB (unknown size counts as 0; the next file is always allowed) and one pool per entity; a config knob or a run-wide pool once a bucket needs more
+PREFETCH_WINDOW = 8
+PREFETCH_BYTES = 512 * 1024 * 1024
+
+
+def localized_inputs(
+    files: Iterable[InputFile], client: Downloader | None, temp_dir: Path
+) -> Generator[LocalInputFile]:
+    """`localize_input` for each file, in order. Close it to stop: pending downloads are cancelled and the running ones awaited."""
+    if client is None:  # local inputs are used in place; no pool
+        for f in files:
+            yield localize_input(f, None, temp_dir)
+        return
+    todo = deque(files)
+    ahead: deque[tuple[Future[LocalInputFile], int]] = deque()
+    pool = ThreadPoolExecutor(PREFETCH_WINDOW, thread_name_prefix="floe-prefetch")
+
+    def refill() -> None:
+        held = sum(size for _, size in ahead)
+        while todo and len(ahead) < PREFETCH_WINDOW:
+            size = todo[0].source_size or 0
+            if ahead and held + size > PREFETCH_BYTES:
+                break
+            held += size
+            ahead.append(
+                (pool.submit(localize_input, todo.popleft(), client, temp_dir), size)
+            )
+
+    try:
+        refill()
+        while ahead:
+            local = ahead.popleft()[0].result()
+            refill()
+            yield local
+    finally:
+        pool.shutdown(cancel_futures=True)

{floe_python-0.8.6 → floe_python-0.8.7}/src/floe/storage/resolve.py

@@ -1,6 +1,7 @@
 """Storage reference resolution: name, else `storages.default`, else local (Rust `StorageResolver::resolve_path`)."""
 
 from pathlib import Path
+from threading import Lock
 
 from floe.config.storage import StorageDefinition, StoragesConfig
 from floe.errors import FloeConfigError, FloeRunError
@@ -48,8 +49,23 @@ def resolve_path(
     return f"s3://{definition.bucket}/{key}" if key else f"s3://{definition.bucket}", definition  # fmt: skip
 
 
+_clients: dict[StorageDefinition, S3Client] = {}
+_clients_lock = Lock()
+
+
+def clear_storage_clients() -> None:
+    """Start of a run: the clients below read credentials and `AWS_*` env when built."""
+    with _clients_lock:
+        _clients.clear()
+
+
 def storage_client(definition: StorageDefinition | None) -> LocalClient | S3Client:
-    """S3 credentials come from the environment, as in Rust."""
+    """S3 credentials come from the environment, as in Rust. One shared client per definition between `clear_storage_clients` calls."""
     if definition is None or definition.type == "local":
         return LocalClient()
-    return S3Client(definition.bucket or "", definition.region, definition.endpoint, definition.path_style_access)  # fmt: skip
+    with (
+        _clients_lock
+    ):  # boto3 clients are thread-safe; the lock only stops a duplicate build
+        if definition not in _clients:
+            _clients[definition] = S3Client(definition.bucket or "", definition.region, definition.endpoint, definition.path_style_access)  # fmt: skip
+        return _clients[definition]

{floe_python-0.8.6 → floe_python-0.8.7}/tests/integration/test_run_seaweedfs.py

@@ -6,6 +6,7 @@ Skipped unless `FLOE_S3_ENDPOINT` is set, e.g. `FLOE_S3_ENDPOINT=http://localhos
 import io
 import json
 import os
+import threading
 import uuid
 from datetime import date
 from pathlib import Path
@@ -15,6 +16,7 @@ import pytest
 
 from floe.config.parse import parse_config
 from floe.run import run
+from floe.storage.prefetch import PREFETCH_WINDOW
 from floe.storage.s3 import S3Client
 
 pytestmark = [
@@ -140,3 +142,29 @@ def test_d30_file_mode_lists_the_s3_source_from_the_window(
     state = s3.read(s3.resolve_uri(f"{prefix}/in/.floe/state/d/orders/state.json"))
     assert state is not None
     assert json.loads(state.body)["watermark"] == "2026-10-24"
+
+
+def test_1102_more_inputs_than_the_prefetch_window_are_each_ingested_once(
+    s3: S3Client, tmp_path: Path
+) -> None:
+    prefix = f"it-{uuid.uuid4().hex}"
+    count = PREFETCH_WINDOW * 2 + 4
+    for i in range(count):
+        s3.write(s3.resolve_uri(f"{prefix}/in/f{i:02}.csv"), f"id\n{i}\n".encode())
+    config = tmp_path / "config.yml"
+    config.write_text(
+        CONFIG.format(
+            bucket=BUCKET, endpoint=os.environ["FLOE_S3_ENDPOINT"], prefix=prefix
+        )
+    )
+    out = run(config, parse_config(config), run_id="r1")
+    assert out.summary["run"]["status"] == "success"
+    part = s3.read(s3.resolve_uri(f"{prefix}/out/accepted/part-00000.parquet"))
+    assert part is not None
+    assert sorted(pl.read_parquet(io.BytesIO(part.body))["id"]) == list(range(count))
+    stored = s3.read(s3.resolve_uri(f"{prefix}/runs/d/run_r1/d/orders/run.json"))
+    assert stored is not None
+    assert [f["input_file"] for f in json.loads(stored.body)["files"]] == [
+        f"s3://{BUCKET}/{prefix}/in/f{i:02}.csv" for i in range(count)
+    ]
+    assert not [t for t in threading.enumerate() if t.name.startswith("floe-prefetch")]

{floe_python-0.8.6 → floe_python-0.8.7}/tests/unit/checks/test_rows.py

@@ -170,6 +170,16 @@ def test_split_rows_without_checks_or_rows() -> None:
     assert accepted.height == rejected.height == 0
 
 
+def test_split_rows_when_no_check_fires_matches_a_rejected_row_schema() -> None:
+    df = pl.DataFrame({"id": ["a", "b"]})
+    nulls = pl.Series([None, None], dtype=pl.Boolean)  # a null mask passes the row
+    passing = [RowCheck("unique", "id", "m", nulls), not_null_check(df, "id")]
+    accepted, rejected = split_rows(df, passing)
+    assert accepted.equals(df) and rejected.height == 0
+    failing = RowCheck("not_null", "id", "m", pl.Series([True, False]))
+    assert rejected.schema == split_rows(df, [failing])[1].schema
+
+
 def test_error_entry_escapes_like_rust() -> None:
     assert error_entry('we"ird\\', "c\n", "m\t\r") == (
         '{"rule":"we\\"ird\\\\","column":"c\\n","message":"m\\t\\r"}'

floe_python-0.8.7/tests/unit/iceberg/test_catalog_reuse.py

@@ -0,0 +1,58 @@
+"""#1101: one opened catalog per `(CatalogDefinition, StorageDefinition)`; the REST catalog is the in-process one of `test_merge_commit_retry.py`."""
+
+import threading
+
+import pytest
+from pyiceberg.catalog import Catalog, load_catalog
+from test_merge_commit_retry import (  # pyright: ignore[reportMissingImports] -- a sibling test module, on pytest's path
+    RestCatalog,
+    server,  # noqa: F401  # pyright: ignore[reportUnusedImport] -- the fixture, used by name
+)
+
+import floe.iceberg.catalog as catalog_module
+from floe.config.storage import CatalogDefinition, StorageDefinition
+from floe.errors import FloeRunError
+from floe.iceberg.catalog import open_catalog
+
+
+@pytest.fixture
+def opened(monkeypatch: pytest.MonkeyPatch) -> list[str]:
+    """The names `load_catalog` was called with; the real call still happens."""
+    calls: list[str] = []
+
+    def counting(name: str, **props: str) -> Catalog:
+        calls.append(name)
+        return load_catalog(name, **props)
+
+    monkeypatch.setattr(catalog_module, "load_catalog", counting)
+    return calls
+
+
+def storage(**values: object) -> StorageDefinition:
+    fields: dict[str, object] = {"name": "s", "type": "s3", "bucket": None, "region": None, "account": None, "container": None, "prefix": None, "endpoint": None, "path_style_access": None}  # fmt: skip
+    return StorageDefinition.model_validate(fields | values)
+
+
+def test_the_same_definition_opens_the_catalog_once(server: tuple[RestCatalog, CatalogDefinition], opened: list[str]) -> None:  # noqa: F811 -- the imported fixture  # fmt: skip
+    _, definition = server
+    assert open_catalog(definition) is open_catalog(definition)
+    assert open_catalog(definition, storage()) is open_catalog(definition, storage())
+    assert len(opened) == 2  # the storage is part of the key
+
+
+def test_concurrent_first_calls_open_it_once(server: tuple[RestCatalog, CatalogDefinition], opened: list[str]) -> None:  # noqa: F811 -- the imported fixture  # fmt: skip
+    _, definition = server
+    threads = [threading.Thread(target=open_catalog, args=(definition,)) for _ in range(8)]  # fmt: skip
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join()
+    assert len(opened) == 1
+
+
+def test_a_failed_open_is_not_kept(opened: list[str]) -> None:
+    broken = CatalogDefinition(name="fake", uri="http://127.0.0.1:1", warehouse_storage=None, warehouse_prefix=None)  # fmt: skip
+    for _ in range(2):
+        with pytest.raises(FloeRunError, match="rest catalog init failed"):
+            open_catalog(broken)
+    assert len(opened) == 2

{floe_python-0.8.6 → floe_python-0.8.7}/tests/unit/run/test_run_entity.py

@@ -520,3 +520,39 @@ def test_d41_records_without_keys_still_face_the_missing_columns_policy(
     )
     assert statuses == ["rejected"]
     assert rows is None or rows.height == 0
+
+
+def test_clean_small_csv_stays_within_the_polars_plan_budget(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """#1104: every eager DataFrame op is a one-node query plan, so each costs about a millisecond."""
+    plans: list[str] = []
+    for name in ("collect", "_collect_eager"):
+        original = vars(pl.LazyFrame)[name]
+
+        def counted(*args: Any, _original: Any = original, **kwargs: Any) -> Any:
+            plans.append("plan")
+            return _original(*args, **kwargs)
+
+        monkeypatch.setattr(pl.LazyFrame, name, counted)
+    rows = "".join(f"{i},n{i}\n" for i in range(200))
+    report, _ = run(tmp_path, "reject", "id,name\n" + rows)
+    assert report["files"][0]["status"] == "success"
+    assert len(plans) <= 6, len(plans)
+
+
+@pytest.mark.parametrize("clash", ["__floe_errors", "_e_cast_id", "__floe_row_index"])
+def test_declared_column_named_like_an_internal_one_is_accepted(
+    tmp_path: Path, clash: str
+) -> None:
+    (tmp_path / "in").mkdir()
+    (tmp_path / "in" / "orders.csv").write_text(f"id,{clash}\n1,hello\n")
+    config = CONFIG.format(severity="reject", mismatch="").replace(
+        "        - {name: name, type: string}",
+        f"        - {{name: {clash}, type: string}}",
+    )
+    (tmp_path / "config.yml").write_text(config)
+    entity = parse_config(tmp_path / "config.yml").entities[0]
+    report = run_entity(entity, tmp_path, "0.3").report
+    assert report["files"][0]["status"] == "success"
+    assert accepted(tmp_path)[clash].to_list() == ["hello"]