floe-python 0.8.5__tar.gz → 0.8.7__tar.gz

{floe_python-0.8.5 → floe_python-0.8.7}/CHANGELOG.md

@@ -2,6 +2,47 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.7
+
+- **The Iceberg catalog is opened once per run (#1101).** Every catalog call used to start a new REST session
+  (a config request, plus an OAuth token request when a credential is set), and an `append` entity did it twice.
+  A run now reuses one catalog per catalog definition and storage, so N entities no longer pay up to 2N
+  initialisations. The catalog is opened again at the start of each run.
+
+- **S3 clients are reused within a run (#1100).** Each `S3Client` builds its own boto3 session, about 55-120 ms,
+  and one S3 entity built 6-8 of them (source, state, accepted, rejected, archive, report). A run now builds one
+  client per storage definition: a two-entity run on one storage went from 5 clients to 1. `AWS_*` variables and
+  credentials are read again at the start of each run, and by `inspect_entity_state()` and
+  `reset_entity_state()`.
+
+- **Remote inputs are downloaded ahead of the file being read (#1102).** Each pending S3 file used to be
+  downloaded only when its turn came, so the read loop waited on every GET. An entity now downloads up to 8
+  files, and about 512 MiB, ahead of the one it is validating (a single larger file still downloads, alone).
+  Files are still processed in the same order, a failed download still raises its own error when that file's turn
+  comes, and local sources are unchanged. With a simulated 0.1 s download per file, 16 files went from 1.72 s to
+  0.24 s.
+
+- **Small CSV files cost fewer Polars plans (#1104).** A clean 200-row file ran 16 query plans between the
+  reader and the writer and now runs 6, with identical output. On 1,200 such files (8 entities of 150 files,
+  Parquet sink on local disk) the run was about 26-28% faster, 7.45 s to 5.38 s in the last measurement.
+
+- **`poll_interval_seconds` and `job_scope` are documented (#1103).** `docs/profiles.md` and
+  `context/orchestrators.md` describe the Kubernetes runner's status poll (default 10 s, a Job that finishes
+  early is noticed at the next poll) and when to choose `job_scope: domain` over `entity`.
+
+## v0.8.6
+
+- **Merge modes keep a key's latest version when it repeats across pending files (#1090).** With `merge_scd1`
+  or `merge_scd2`, a run that picks up several batches used to keep the oldest file's row and reject the newer
+  ones. A key repeated across files now resolves to the latest file in file order, and the older rows are
+  rejected as duplicates. Within one file the first row still wins. Under `policy.severity: abort` the order is
+  unchanged.
+
+- **`--full-refresh` rebuilds a `merge_scd1` table from the inputs (#1091).** It used to be rejected. It now reads
+  every input and replaces the table with their rows in one commit, so a row deleted at the source leaves the
+  table. Merge-key checks still apply, and a key in several inputs keeps the latest file's row. `merge_scd2` still
+  rejects the flag, and the message now says its version history cannot be rebuilt from the inputs.
+
 ## v0.8.5
 
 - **An output column name collision now fails `floe validate`, not the run (#1080).** A column with an explicit
@@ -55,6 +96,14 @@ All notable changes to Floe are documented in this file.
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in
   dagster-floe: `args` now prefix Floe's rendered arguments instead of replacing them.
 
+## dagster-floe v0.4.3
+
+- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
+  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
+  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
+  `merge_scd1` entity needs Floe 0.8.6 or newer, which rebuilds the table from the inputs; `merge_scd2`
+  still rejects `--full-refresh`.
+
 ## dagster-floe v0.4.2
 
 - **Silver lineage events from a `kubernetes_job` run now carry the Dagster parent run (#1076).** The Job

{floe_python-0.8.5 → floe_python-0.8.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.5
+Version: 0.8.7
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.5 → floe_python-0.8.7}/README.md

@@ -8,7 +8,8 @@ layer, routing accepted rows to lakehouse sinks and rejected rows to quarantine
 with audit reports.
 
 Use Floe when you already have a platform such as an Iceberg lakehouse on
-Polaris or Snowflake Open Catalog, run by Dagster or Airflow, but you need a lightweight entry gate for file contracts, quality checks, rejected rows,
+Polaris or Snowflake Open Catalog (with static storage credentials: Floe does not use
+catalog-vended credentials), run by Dagster or Airflow, but you need a lightweight entry gate for file contracts, quality checks, rejected rows,
 and run evidence.
 
 Floe complements extract/load tools such as dlt, ingestr, and Airbyte: they get
@@ -39,8 +40,9 @@ Each `floe run` executes a deterministic gate per entity:
 | **5. Write outputs** | Route valid rows to accepted sinks, invalid rows to rejected sinks, and write reports |
 
 Floe is a Python package. It reads and validates with Polars, writes Iceberg
-tables through PyIceberg, and runs Iceberg merges and unique checks against
-table history through DuckDB, all on a single node.
+tables through PyIceberg, and runs Iceberg merges through DuckDB, all on a
+single node. Unique checks against table history semi-join each chunk with Polars
+against a local cache of the table's key columns.
 
 - **Inputs:** CSV · TSV · JSON · Parquet · ORC · Avro · XLSX · XML
 - **Accepted outputs:** Parquet · Apache Iceberg

{floe_python-0.8.5 → floe_python-0.8.7}/context/decisions/polars-and-arrow.md

@@ -15,8 +15,8 @@ own data model.
 
 Use **Polars** as the single DataFrame engine for reading and validation. Data crosses to the
 writers as **Apache Arrow** through the Arrow C Data Interface: Polars → pyarrow → PyIceberg for
-overwrite and append, and Arrow / staged Parquet → DuckDB for merges and unique checks against
-table history.
+overwrite and append, and Arrow / staged Parquet → DuckDB for merges. Unique checks against
+table history stay in Polars (a semi-join against a local cache of the table's key columns).
 
 ## Rationale
 

{floe_python-0.8.5 → floe_python-0.8.7}/context/decisions/python-core-for-olf.md

@@ -44,8 +44,9 @@ gates below pass.
 - **Polars (Python)** reads and validates. It is the same Rust engine, so the column operations
   run at the same speed.
 - **PyIceberg** owns the Polaris catalog, schema evolution and plain commits.
-- **DuckDB**, attached to Polaris, does anything that touches the target table at scale: MERGE,
-  and unique checks against history. It spills to disk.
+- **DuckDB**, attached to Polaris, does anything that touches the target table at scale: MERGE.
+  It spills to disk. (Unique checks against history moved to Polars in #1057; see the note under
+  Data paths.)
 - Python's Arrow libraries exchange data through the Arrow C Data Interface, which doesn't depend
   on library versions. The lockstep upgrade problem goes away.
 

{floe_python-0.8.5 → floe_python-0.8.7}/context/orchestrators.md

@@ -211,6 +211,7 @@ When `type = kubernetes_job`, the manifest runner definition supports:
 | `node_selector` | object | Pod `nodeSelector` |
 | `tolerations` | array | Pod `tolerations`, verbatim |
 | `backoff_limit` | integer >= 0 | Job `backoffLimit` (default `0`) |
+| `poll_interval_seconds` | integer >= 1 | Seconds between status reads of the Job (default `10`) |
 | `env_from` | array | Container `envFrom`, verbatim, e.g. `[{secretRef: {name: floe-env}}]` |
 
 These fields are populated automatically when `floe manifest generate` is given a profile with a `kubernetes_job` runner block. The `dagster-floe[kubernetes]` optional extra expresses the Kubernetes client requirement.
@@ -220,6 +221,8 @@ These fields are populated automatically when `floe manifest generate` is given
 - **`entity`** (default, also when unset): one Job per entity, named `floe-<entity>-<id>`. Pod start is paid once per entity; each entity has its own timeout, TTL and failure. Use it for a few large entities, or when entities need isolation.
 - **`domain`**: entities of one domain whose runner has `job_scope: domain` form one subsettable multi-asset per (domain, runner), so an entity with another runner gets its own Job. Materializing it launches **one** Job named `floe-<domain>-<id>` running `floe run --manifest <uri> --entities a,b,c` (`--domain <d>` when they are every entity of the domain in the manifest) for the selected entities in manifest order, one after another, so Polaris never sees overlapping commits from the Job. The Job is sized from `domain_resources[<domain>]`, else `resources`; `timeout_seconds`, TTL and polling apply to the one Job. Use it when pod start dominates (many small entities). Select the accepted assets: a rejected asset is materialized only when it is selected too, and the asset job selects both. The `floe/table` op tag (#924) is not set on a domain asset.
 
+**Choosing a scope and a poll interval.** The runner reads the Job's status every `poll_interval_seconds`, so a Job that finishes early is only noticed at the next poll: with the default 10 s, a 3 s Job is seen about 7 s late, a flat per-Job cost. `entity` pays pod start and that polling delay once per entity, but entities are isolated and run in parallel through Dagster's executor. `domain` pays both once per domain, but runs its entities one after another. For short Jobs, lower `poll_interval_seconds` (for example `2`), at the cost of more Kubernetes API reads.
+
 Dagster still reports per entity (materialization, rows, rejected counts, report URI, asset checks) from the run's `run_finished` event and summary.
 
 **After a failure** each entity is reported from its own status in the run summary: `rejected` and `success_with_warnings` entities are materialized, an `aborted` / `failed` entity fails, and an entity absent from the summary never ran and is reported as *not run* (listed in the step failure's `not_run_entities`), not as its own failure. Since D36 (#959) a policy `severity: abort` fails only its entity and the run continues with the next one. A global failure (access, memory, any raised `FloeError`) still aborts the run. Re-executing the step reruns the selected entities; subset it to rerun only some. Report URIs come from `run_finished` (`summary_uri`, `entity_report_uris`), never built by hand (#960).

{floe_python-0.8.5 → floe_python-0.8.7}/docs/cli.md

@@ -25,7 +25,9 @@
     An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state
     already lists are read again, `append` writes as `overwrite`, and with `sink.archive` the archived files are read
-    too (D24). It is rejected with merge modes.
+    too (D24). `merge_scd1` replaces the table with the inputs' rows in one commit, so a row deleted at the source
+    leaves the table; a key in several inputs keeps the latest file's row. It is rejected with `merge_scd2`, whose
+    version history cannot be rebuilt from the inputs.
   - `--log-format json` emits NDJSON events to stdout (useful for orchestrators); `off` is the default.
   - `--dry-run` resolves entity inputs and output targets without running ingestion.
   - `--quiet` reduces console output to run totals.

{floe_python-0.8.5 → floe_python-0.8.7}/docs/how-it-works.md

@@ -68,6 +68,10 @@ For files that passed prechecks:
 
 After row-level validation, uniqueness is evaluated across the **entire entity**
 in file order. The first occurrence is accepted; later duplicates are flagged.
+With `merge_scd1` and `merge_scd2`, a key repeated across files keeps the
+occurrence from the latest file (#1090), so a run that catches up on several
+batches merges each key's newest version; within one file the first occurrence
+still wins.
 With `append`, keys already in the accepted sink count as seen,
 so a row that duplicates the existing history is flagged too.
 

{floe_python-0.8.5 → floe_python-0.8.7}/docs/installation.md

@@ -1,6 +1,7 @@
 # Installation
 
-Floe is a pure-Python package (`floe-python` on PyPI) that installs the `floe` command and the
+Floe is a pure-Python package (`floe-python` on PyPI; the `floe` name there belongs to an
+unrelated project) that installs the `floe` command and the
 `floe` Python library. It is also published as a Docker image.
 
 ## pip (all platforms)

{floe_python-0.8.5 → floe_python-0.8.7}/docs/profiles.md

@@ -238,12 +238,22 @@ execution:
     node_selector: {pool: batch}
     tolerations: [{key: batch, operator: Exists, effect: NoSchedule}]   # verbatim
     backoff_limit: 2                         # Job backoffLimit, default 0
+    poll_interval_seconds: 2                 # seconds between Job status reads, default 10
     env_from: [{secretRef: {name: floe-env}}]   # container envFrom, verbatim
 ```
 
 `labels`, `annotations` and `node_selector` are string maps, `backoff_limit` is a non-negative integer,
 and `security_context`, `tolerations` and `env_from` are copied as written: Kubernetes validates them.
 
+`poll_interval_seconds` is an integer of at least 1 (default 10): dagster-floe reads the Job's status that
+often, so a Job that finishes early is only noticed at the next poll. A lower value such as `2` shortens
+short Jobs at the cost of more Kubernetes API reads.
+
+`job_scope` (`entity` by default, or `domain`) sets how many Jobs a run launches. `entity` starts one Job
+per entity: each pays pod start and one poll interval, but they are isolated and run in parallel through
+Dagster's executor. `domain` starts one Job per domain: pod start and polling are paid once, but its
+entities run one after another.
+
 ---
 
 ## Using profile variables in a Floe config

{floe_python-0.8.5 → floe_python-0.8.7}/docs/summary.md

@@ -13,7 +13,8 @@ need a lightweight entry gate for file contracts and quality checks.
 - Raw exports, partner feeds, API extracts, or object-storage drops need schema
   and quality validation before becoming trusted data.
 - Existing platforms such as an Iceberg lakehouse on Polaris or Snowflake Open
-  Catalog, run by Dagster or Airflow, need an embeddable ingestion gate rather
+  Catalog (with static storage credentials: Floe does not use catalog-vended
+  credentials), run by Dagster or Airflow, need an embeddable ingestion gate rather
   than another platform.
 - Extract/load tools such as dlt, ingestr, or Airbyte already move data, and
   Floe should validate and route the landed files or datasets.

{floe_python-0.8.5 → floe_python-0.8.7}/docs/write_modes.md

@@ -30,6 +30,9 @@ Each entity's accepted rows land in one Iceberg catalog commit.
 - `schema.primary_key` is required and is used as merge key.
 - Source rows are validated for merge-key uniqueness during row checks.
   In `policy.severity=warn`, duplicate merge-key rows are rejected before merge so only unambiguous rows are merged.
+  A key repeated across the run's input files keeps the row from the latest file (in file order);
+  the older rows are rejected as duplicates. Within one file the first row wins. The same holds
+  for `merge_scd2`. Under `policy.severity: abort` a repeated key aborts the later file instead.
 - On key match: update non-key columns from source, only when a value changed, so rerunning a
   batch writes nothing.
 - `sink.accepted.merge.ignore_columns` can exclude additional business columns from SCD1 update sets.
@@ -41,7 +44,8 @@ Each entity's accepted rows land in one Iceberg catalog commit.
 - Strict schema handling remains the default behavior.
 - `schema.schema_evolution.mode: add_columns` applies as for standard writes; merge-key columns
   cannot be introduced by schema evolution.
-- `--full-refresh` is rejected with merge modes.
+- `--full-refresh` replaces the table with the inputs' rows in one commit (written as `overwrite`, with the
+  merge-key checks above), so rows deleted at the source leave the table. `merge_scd2` rejects it.
 - Merge-on-read leaves position delete files; `floe maintain` expires old snapshots and removes
   orphan files (see [CLI](cli.md)).
 

{floe_python-0.8.5 → floe_python-0.8.7}/orchestrators/dagster-floe/README.md

@@ -42,6 +42,7 @@ pip install "dagster-floe[floe]"   # also installs floe-python>=0.8.3,<0.9 for t
 
 | dagster-floe | Floe | Manifest `spec_version` |
 | --- | --- | --- |
+| 0.4.3 | 0.8.3 or newer (the `floe/full_refresh` run tag on a `merge_scd1` entity needs 0.8.6) | `0.8` |
 | 0.4.2 | 0.8.3 or newer | `0.8` |
 | 0.4.1 | 0.8.3 or newer (`kubernetes_job` runs pass the profile as `-p env:FLOE_PROFILE_CONTENT`) | `0.8` |
 | 0.4.0 | 0.8.1 or newer (a Glue `{domain}` database needs 0.8.2) | `0.8` |
@@ -189,6 +190,13 @@ floe manifest generate -c orders.yml -p prod.yml --output manifests/orders.json
 
 See [docs/lineage.md](../../docs/lineage.md) for the full lineage config reference.
 
+## Full refresh for one run
+
+Tag a Dagster run `floe/full_refresh=true` (in the Launchpad, with `dagster job launch --tags '{"floe/full_refresh": "true"}'`,
+or in a schedule's `tags`) and every entity it materializes runs `floe run ... --full-refresh`, whatever the runner
+(in-process, `local_process`, `kubernetes_job`, `databricks_job`, either `job_scope`). Runs without the tag
+are unchanged. See [docs/cli.md](../../docs/cli.md) for what `--full-refresh` does in each write mode.
+
 ## Output metadata
 
 Every accepted and rejected asset carries `floe/entity` (the manifest entity name) and `floe/output` (`"accepted"` or `"rejected"`) in its asset metadata, so tooling can tell the two apart without parsing dagster-floe's output names: `spec.metadata["floe/output"]` on each `AssetSpec` of `multi_asset.specs`.

{floe_python-0.8.5 → floe_python-0.8.7}/orchestrators/dagster-floe/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dagster-floe"
-version = "0.4.2"
+version = "0.4.3"
 description = "Dagster connector for Floe CLI (config-driven ingestion)"
 requires-python = ">=3.10"
 readme = "README.md"

{floe_python-0.8.5 → floe_python-0.8.7}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -41,6 +41,8 @@ from .runner import RunResult, Runner
 # Advisory Dagster tag (#924): entities writing one Iceberg table share its value, so a
 # `tag_concurrency_limits` entry on this key keeps their runs apart.
 TABLE_TAG = "floe/table"
+# Run tag (#1091): `floe/full_refresh=true` runs this Dagster run's entities with `floe run --full-refresh`.
+FULL_REFRESH_TAG = "floe/full_refresh"
 
 SourceKeyFn = Callable[[ManifestEntity], AssetKey | None]
 
@@ -320,7 +322,7 @@ def _make_entity_multi_asset(
             run_id=run_id,
             entity=entity_name,
             log_format=execution.log_format,
-            execution=execution,
+            execution=_for_run(execution, run),
             runner_definition=runner_definition,
             dagster_job_name=dagster_job_name,
         )
@@ -347,6 +349,14 @@ def _make_entity_multi_asset(
     return _multi_asset
 
 
+def _for_run(execution: ManifestExecution, run: Any) -> ManifestExecution:
+    """`execution`, with `--full-refresh` appended when the Dagster run carries the full refresh tag."""
+    tags = getattr(run, "tags", None) or {}
+    if tags.get(FULL_REFRESH_TAG) != "true" or "--full-refresh" in [*execution.base_args, *execution.per_entity_args]:
+        return execution
+    return replace(execution, per_entity_args=[*execution.per_entity_args, "--full-refresh"])
+
+
 def _parse_finished(context: Any, result: RunResult, execution: ManifestExecution, run_id: str | None):
     try:
         events = parse_json_event_lines(result.stdout)
@@ -574,7 +584,9 @@ def _make_domain_multi_asset(
             run_id=floe_run_id,
             domain=domain,
             entities=names,
-            execution=replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution,
+            execution=_for_run(
+                replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution, run
+            ),
             runner_definition=runner_definition,
             dagster_run_id=run_id,
             dagster_job_name=dagster_job_name,

{floe_python-0.8.5 → floe_python-0.8.7}/orchestrators/dagster-floe/tests/test_config_mode.py

@@ -38,7 +38,7 @@ def paths(tmp_path: Path) -> tuple[str, str, str]:
     return str(config), str(profile), str(manifest)
 
 
-def _k8s_args(**source: str) -> tuple[list[str], list[list[str]]]:
+def _k8s_args(tags: dict[str, str] | None = None, **source: str) -> tuple[list[str], list[list[str]]]:
     """Asset keys, and the command each materialized entity's Kubernetes Job runs."""
     calls: list[list[str]] = []
 
@@ -49,7 +49,7 @@ def _k8s_args(**source: str) -> tuple[list[str], list[list[str]]]:
     defs = load_floe_assets(**source, runner=LocalRunner(), domains=["sales"])
     assets = list(defs.assets or [])
     with patch("floe_dagster.kubernetes_runner.run_kubernetes_job", job):
-        materialize(assets, raise_on_error=False)
+        materialize(assets, raise_on_error=False, tags=tags)
     keys = sorted(k.to_user_string() for a in assets for k in ([a.key] if isinstance(a, SourceAsset) else a.keys))
     return keys, sorted(calls)
 
@@ -99,6 +99,33 @@ def test_in_process_runner_runs_from_the_config_and_profile(paths) -> None:
     assert sorted(calls, key=str) == expected
 
 
+def test_full_refresh_run_tag_adds_the_flag_to_kubernetes_job_runs(paths) -> None:
+    _, _, manifest = paths
+    tail = ["--log-format", "json", "--quiet", "--entities"]
+    _, tagged = _k8s_args(tags={"floe/full_refresh": "true"}, manifest_path=manifest)
+    _, untagged = _k8s_args(manifest_path=manifest)
+    assert tagged == [["floe", "run", "--manifest", manifest, *tail, e, "--full-refresh"] for e in ("invoices", "orders")]
+    assert untagged == [["floe", "run", "--manifest", manifest, *tail, e] for e in ("invoices", "orders")]
+
+
+def test_full_refresh_run_tag_runs_in_process_with_full_refresh(paths) -> None:
+    config, _, _ = paths
+    calls: list[object] = []
+
+    class Outcome:
+        summary = {"run": {"exit_code": 0}}
+
+    def run(config_path: str, **kwargs: object) -> Outcome:
+        calls.append(kwargs["full_refresh"])
+        return Outcome()
+
+    defs = load_floe_assets(config_path=config, runner=InProcessRunner(), domains=["sales"])
+    with patch.object(floe, "run", run):
+        materialize(list(defs.assets or []), raise_on_error=False, tags={"floe/full_refresh": "true"})
+        materialize(list(defs.assets or []), raise_on_error=False)
+    assert calls == [True, True, False, False]
+
+
 def _domain_op_names(config: str, profile: str, **kwargs: str) -> set[str]:
     defs = load_floe_assets(config_path=config, profile_path=profile, runner=LocalRunner(), **kwargs)
     return {a.op.name for a in defs.assets or [] if not isinstance(a, SourceAsset) and a.op.name.startswith("floe_")}

{floe_python-0.8.5 → floe_python-0.8.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.5"
+version = "0.8.7"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/__init__.py

@@ -46,6 +46,7 @@ from floe.run.events import (
 )
 from floe.state import dump_state, read_state, reset
 from floe.storage.paths import normalize_local_path
+from floe.storage.resolve import clear_storage_clients
 from floe.storage.types import StorageClient
 
 _observer: Callable[[RunEvent], None] | None = None
@@ -281,6 +282,7 @@ def extract_config_env_vars(config_path: str) -> dict[str, str]:
 
 
 def _state(config_path: str, entity_name: str) -> tuple[str, str, str, StorageClient]:
+    clear_storage_clients()
     path, root = _load(config_path)
     [entity] = _run.select_entities(root, [entity_name])
     uri, client = state_uri(entity, normalize_local_path(path.parent), root.storages)

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/checks/normalize.py

@@ -115,4 +115,6 @@ def normalize_columns(df: pl.DataFrame, strategy: str | None) -> pl.DataFrame:
                 f"normalized input column collision: {seen[new]} and {old} -> {new}"
             )
         seen[new] = old
-    return df.rename(names)
+    renamed = df.clone()
+    renamed.columns = list(names.values())  # a rename is a query plan, this is not
+    return renamed

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/checks/rows.py

@@ -109,6 +109,10 @@ def split_rows(
     sources: Mapping[str, str] | None = None,
 ) -> tuple[pl.DataFrame, pl.DataFrame]:
     """`(accepted, rejected)`; rejected adds `__floe_row_index` (UInt64, row position in `df`) and `__floe_errors`."""
+    # a null mask passes the row, as in `error_column`
+    if not any(c.mask.any() for c in checks):
+        none = [pl.Series(ROW_INDEX, [], pl.UInt64), pl.Series(ERRORS, [], pl.String)]
+        return df, df.clear().with_columns(none)  # replaces a clashing column, as below
     errors = error_column(df.height, checks, formatter, sources)
     rejected = df.with_columns(
         pl.int_range(pl.len(), dtype=pl.UInt64).alias(ROW_INDEX), errors

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/iceberg/catalog.py

@@ -4,6 +4,7 @@ import os
 import re
 import shutil
 import tempfile
+import threading
 import weakref
 from collections.abc import Generator, Sequence
 from contextlib import contextmanager
@@ -83,8 +84,34 @@ def _credential_props(catalog: str, credential: str) -> dict[str, str]:
     return {"credential": credential.removeprefix("client_credentials:")}
 
 
+_CATALOGS: dict[tuple[CatalogDefinition, StorageDefinition | None], Catalog] = {}
+_CATALOGS_LOCK = threading.Lock()
+
+
+def clear_catalogs() -> None:
+    """Called when a run starts: a catalog is reused within a run only, never across runs (#1101)."""
+    with _CATALOGS_LOCK:
+        _CATALOGS.clear()
+
+
 def open_catalog(
     definition: CatalogDefinition, storage: StorageDefinition | None = None
+) -> Catalog:
+    """One catalog per definition and storage until `clear_catalogs`, so the REST init (config, OAuth token) happens once.
+
+    Safe to call from several threads (one init per key). PyIceberg 0.12 refetches an expired OAuth token
+    itself: its REST calls retry once on a 401 after a new client-credentials request; a static `token:` has
+    nothing to refetch, as with a freshly opened catalog. A failed init is not kept.
+    """
+    key = (definition, storage)
+    with _CATALOGS_LOCK:
+        if key not in _CATALOGS:
+            _CATALOGS[key] = _open_catalog(definition, storage)
+        return _CATALOGS[key]
+
+
+def _open_catalog(
+    definition: CatalogDefinition, storage: StorageDefinition | None
 ) -> Catalog:
     """Credentials for S3 and Glue come from the environment; the catalog never vends them (as in Rust)."""
     props = {

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/iceberg/sink.py

@@ -177,7 +177,8 @@ def write_iceberg(
     key = [output.get(k.strip(), k.strip()) for k in primary_key]
     identifier = (target.namespace, target.table)
     with run_error("rest catalog table_exists check failed"):
-        exists = mode in ("merge_scd1", "merge_scd2") and catalog.table_exists(identifier)  # fmt: skip
+        # the configured mode: a refreshed merge writes as overwrite and keeps the key guard (#1091)
+        exists = entity.sink.write_mode in ("merge_scd1", "merge_scd2") and catalog.table_exists(identifier)  # fmt: skip
     if exists:
         with run_error("rest catalog load_table failed"):
             existing = set(catalog.load_table(identifier).schema().column_names)

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/read/csv.py

@@ -79,15 +79,16 @@ def _scan_args(options: SourceOptions) -> _Args:
 
 def _columns(path: Path, args: _Args, declared: Sequence[str]) -> list[str]:
     try:
-        probe = args.scan(path, n_rows=0 if args.has_header else 1).collect()
+        probe = args.scan(path, n_rows=0 if args.has_header else 1)
+        if args.has_header:  # the schema is the header: no plan to run
+            return probe.collect_schema().names()
+        width = probe.collect().width
     except (pl.exceptions.PolarsError, OSError) as err:
         # Rust's polars prints the error kind: `no data: empty CSV`
         kind = "no data: " if isinstance(err, pl.exceptions.NoDataError) else ""
         raise FloeIoError(f"csv header read failed: {kind}{err}") from err
-    if args.has_header:
-        return probe.columns
-    names = list(declared[: probe.width])
-    return names + [f"extra_column_{i + 1}" for i in range(len(declared), probe.width)]
+    names = list(declared[:width])
+    return names + [f"extra_column_{i + 1}" for i in range(len(declared), width)]
 
 
 def _field_counts(path: Path, args: _Args, skip: int) -> Iterator[int]:

{floe_python-0.8.5 → floe_python-0.8.7}/src/floe/run/__init__.py

@@ -11,6 +11,7 @@ from floe.checks.rows import Formatter
 from floe.config.project import RootConfig
 from floe.config.types import EntityConfig
 from floe.errors import FloeConfigError, FloeError
+from floe.iceberg.catalog import clear_catalogs
 from floe.report.summary import (
     build_summary,
     id_path,
@@ -24,7 +25,7 @@ from floe.run.entity import EntityRun, list_source, resolve_source, run_entity
 from floe.run.events import Emitter, Observer
 from floe.storage.inputs import ResolvedInputs
 from floe.storage.paths import normalize_local_path
-from floe.storage.resolve import resolve_path, storage_client
+from floe.storage.resolve import clear_storage_clients, resolve_path, storage_client
 
 
 @dataclass(frozen=True)
@@ -72,10 +73,10 @@ def _rfc3339() -> str:
 
 
 def _refresh(entity: EntityConfig) -> EntityConfig:
-    """Full refresh: an append sink overwrites; merge sinks cannot refresh."""
+    """Full refresh: an append sink overwrites; `merge_scd1` overwrites in `run_entity`, which keeps its merge checks."""
     mode = entity.sink.write_mode
-    if mode in ("merge_scd1", "merge_scd2"):
-        raise FloeConfigError(f"entity '{entity.name}': --full-refresh is not supported with write_mode '{mode}'")  # fmt: skip
+    if mode == "merge_scd2":
+        raise FloeConfigError(f"entity '{entity.name}': --full-refresh is not supported with write_mode '{mode}': its version history cannot be rebuilt from the inputs")  # fmt: skip
     if mode != "append":
         return entity
     return entity.model_copy(update={"sink": entity.sink.model_copy(update={"write_mode": "overwrite"})})  # fmt: skip
@@ -104,7 +105,9 @@ def run(
     config_uri: str | None = None,
 ) -> RunOutcome:
     """Run the selected entities in config order, writing each entity's report as it finishes. A failing entity raises: no summary, as Rust."""
+    clear_storage_clients()
     started, timer = _rfc3339(), time.monotonic()
+    clear_catalogs()
     selected = select_entities(root, entities)
     if full_refresh:
         selected = [_refresh(e) for e in selected]