floe-python 0.8.4__tar.gz → 0.8.6__tar.gz

{floe_python-0.8.4 → floe_python-0.8.6}/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 All notable changes to Floe are documented in this file.
 
+## v0.8.6
+
+- **Merge modes keep a key's latest version when it repeats across pending files (#1090).** With `merge_scd1`
+  or `merge_scd2`, a run that picks up several batches used to keep the oldest file's row and reject the newer
+  ones. A key repeated across files now resolves to the latest file in file order, and the older rows are
+  rejected as duplicates. Within one file the first row still wins. Under `policy.severity: abort` the order is
+  unchanged.
+
+- **`--full-refresh` rebuilds a `merge_scd1` table from the inputs (#1091).** It used to be rejected. It now reads
+  every input and replaces the table with their rows in one commit, so a row deleted at the source leaves the
+  table. Merge-key checks still apply, and a key in several inputs keeps the latest file's row. `merge_scd2` still
+  rejects the flag, and the message now says its version history cannot be rebuilt from the inputs.
+
+## v0.8.5
+
+- **An output column name collision now fails `floe validate`, not the run (#1080).** A column with an explicit
+  `source` keeps its `name`; a column without one is normalised. When the two end on the same output name,
+  `floe validate` passed and `floe run` then failed. Validate now rejects it, and the message names the two
+  columns as written instead of their normalised sources.
+
 ## v0.8.4
 
 - **Columns that normalise to the same name now fail `floe validate`, not the run (#1005).** With
@@ -43,6 +63,13 @@ All notable changes to Floe are documented in this file.
 
 ## Unreleased
 
+### dagster-floe
+
+- **Per-run full refresh from the `floe/full_refresh` run tag (#1091).** A Dagster run tagged
+  `floe/full_refresh=true` runs its entities with `floe run --full-refresh`, in process or through a
+  `local_process`, `kubernetes_job` or `databricks_job` runner. Runs without the tag are unchanged. A
+  `merge_scd1` / `merge_scd2` entity needs a Floe version that accepts `--full-refresh` with merge modes.
+
 ### airflow-floe
 
 - **A `kubernetes_job` runner `args` override no longer drops the entity selection (#1031)**, as in

{floe_python-0.8.4 → floe_python-0.8.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: floe-python
-Version: 0.8.4
+Version: 0.8.6
 Summary: Floe: a data-contract runtime for file ingestion
 License-File: LICENSE
 Requires-Python: >=3.11

{floe_python-0.8.4 → floe_python-0.8.6}/README.md

@@ -8,7 +8,8 @@ layer, routing accepted rows to lakehouse sinks and rejected rows to quarantine
 with audit reports.
 
 Use Floe when you already have a platform such as an Iceberg lakehouse on
-Polaris or Snowflake Open Catalog, run by Dagster or Airflow, but you need a lightweight entry gate for file contracts, quality checks, rejected rows,
+Polaris or Snowflake Open Catalog (with static storage credentials: Floe does not use
+catalog-vended credentials), run by Dagster or Airflow, but you need a lightweight entry gate for file contracts, quality checks, rejected rows,
 and run evidence.
 
 Floe complements extract/load tools such as dlt, ingestr, and Airbyte: they get
@@ -39,8 +40,9 @@ Each `floe run` executes a deterministic gate per entity:
 | **5. Write outputs** | Route valid rows to accepted sinks, invalid rows to rejected sinks, and write reports |
 
 Floe is a Python package. It reads and validates with Polars, writes Iceberg
-tables through PyIceberg, and runs Iceberg merges and unique checks against
-table history through DuckDB, all on a single node.
+tables through PyIceberg, and runs Iceberg merges through DuckDB, all on a
+single node. Unique checks against table history semi-join each chunk with Polars
+against a local cache of the table's key columns.
 
 - **Inputs:** CSV · TSV · JSON · Parquet · ORC · Avro · XLSX · XML
 - **Accepted outputs:** Parquet · Apache Iceberg

{floe_python-0.8.4 → floe_python-0.8.6}/context/coding-standards.md

@@ -32,7 +32,8 @@ Context: `context/architecture.md` and `context/decisions/python-core-for-olf.md
 
 - No per-row Python loops on data. Use Polars or DuckDB expressions.
 - Keep memory bounded by the chunk or the batch, not the table: read in chunks, and let DuckDB
-  do anything that scans the target table (merges, unique checks against history).
+  do the merges. Unique checks against history semi-join each chunk with Polars against a local
+  cache of the target's key columns.
 
 ## Comments
 

{floe_python-0.8.4 → floe_python-0.8.6}/context/decisions/polars-and-arrow.md

@@ -15,8 +15,8 @@ own data model.
 
 Use **Polars** as the single DataFrame engine for reading and validation. Data crosses to the
 writers as **Apache Arrow** through the Arrow C Data Interface: Polars → pyarrow → PyIceberg for
-overwrite and append, and Arrow / staged Parquet → DuckDB for merges and unique checks against
-table history.
+overwrite and append, and Arrow / staged Parquet → DuckDB for merges. Unique checks against
+table history stay in Polars (a semi-join against a local cache of the table's key columns).
 
 ## Rationale
 

{floe_python-0.8.4 → floe_python-0.8.6}/context/decisions/python-core-for-olf.md

@@ -44,8 +44,9 @@ gates below pass.
 - **Polars (Python)** reads and validates. It is the same Rust engine, so the column operations
   run at the same speed.
 - **PyIceberg** owns the Polaris catalog, schema evolution and plain commits.
-- **DuckDB**, attached to Polaris, does anything that touches the target table at scale: MERGE,
-  and unique checks against history. It spills to disk.
+- **DuckDB**, attached to Polaris, does anything that touches the target table at scale: MERGE.
+  It spills to disk. (Unique checks against history moved to Polars in #1057; see the note under
+  Data paths.)
 - Python's Arrow libraries exchange data through the Arrow C Data Interface, which doesn't depend
   on library versions. The lockstep upgrade problem goes away.
 
@@ -69,6 +70,11 @@ gates below pass.
 | `merge_scd1` / `merge_scd2` | Chunks staged as Parquet, then one DuckDB `MERGE INTO` (merge-on-read, positional deletes). SCD2 sends each changed row twice: once to close the current version, once to insert the new one. | Spills to disk |
 | `unique` vs target history | DuckDB anti-join between staged keys and the table's key columns | Spills to disk |
 
+> **Note (2026-10-29, #1084):** the `unique` row above is no longer how the check runs. Since #1057
+> the target's key columns are cached locally once per entity, and each chunk semi-joins against that
+> cache with Polars (`src/floe/iceberg/unique.py`). DuckDB runs the merges only. The G6 result below
+> was measured with the DuckDB anti-join.
+
 Re-running an SCD1 merge with the same batch leaves the table unchanged, and an SCD2 batch with
 no changes adds no versions. Only `append` needs the retry guard.
 

{floe_python-0.8.4 → floe_python-0.8.6}/docs/cli.md

@@ -25,7 +25,9 @@
     An unknown domain is an error, and `--domain` cannot be combined with `--entities`.
   - `-F` / `--full-refresh` rebuilds the accepted output from every input: files the incremental state
     already lists are read again, `append` writes as `overwrite`, and with `sink.archive` the archived files are read
-    too (D24). It is rejected with merge modes.
+    too (D24). `merge_scd1` replaces the table with the inputs' rows in one commit, so a row deleted at the source
+    leaves the table; a key in several inputs keeps the latest file's row. It is rejected with `merge_scd2`, whose
+    version history cannot be rebuilt from the inputs.
   - `--log-format json` emits NDJSON events to stdout (useful for orchestrators); `off` is the default.
   - `--dry-run` resolves entity inputs and output targets without running ingestion.
   - `--quiet` reduces console output to run totals.

{floe_python-0.8.4 → floe_python-0.8.6}/docs/how-it-works.md

@@ -68,6 +68,10 @@ For files that passed prechecks:
 
 After row-level validation, uniqueness is evaluated across the **entire entity**
 in file order. The first occurrence is accepted; later duplicates are flagged.
+With `merge_scd1` and `merge_scd2`, a key repeated across files keeps the
+occurrence from the latest file (#1090), so a run that catches up on several
+batches merges each key's newest version; within one file the first occurrence
+still wins.
 With `append`, keys already in the accepted sink count as seen,
 so a row that duplicates the existing history is flagged too.
 

{floe_python-0.8.4 → floe_python-0.8.6}/docs/installation.md

@@ -1,6 +1,7 @@
 # Installation
 
-Floe is a pure-Python package (`floe-python` on PyPI) that installs the `floe` command and the
+Floe is a pure-Python package (`floe-python` on PyPI; the `floe` name there belongs to an
+unrelated project) that installs the `floe` command and the
 `floe` Python library. It is also published as a Docker image.
 
 ## pip (all platforms)

{floe_python-0.8.4 → floe_python-0.8.6}/docs/summary.md

@@ -13,7 +13,8 @@ need a lightweight entry gate for file contracts and quality checks.
 - Raw exports, partner feeds, API extracts, or object-storage drops need schema
   and quality validation before becoming trusted data.
 - Existing platforms such as an Iceberg lakehouse on Polaris or Snowflake Open
-  Catalog, run by Dagster or Airflow, need an embeddable ingestion gate rather
+  Catalog (with static storage credentials: Floe does not use catalog-vended
+  credentials), run by Dagster or Airflow, need an embeddable ingestion gate rather
   than another platform.
 - Extract/load tools such as dlt, ingestr, or Airbyte already move data, and
   Floe should validate and route the landed files or datasets.

{floe_python-0.8.4 → floe_python-0.8.6}/docs/write_modes.md

@@ -30,6 +30,9 @@ Each entity's accepted rows land in one Iceberg catalog commit.
 - `schema.primary_key` is required and is used as merge key.
 - Source rows are validated for merge-key uniqueness during row checks.
   In `policy.severity=warn`, duplicate merge-key rows are rejected before merge so only unambiguous rows are merged.
+  A key repeated across the run's input files keeps the row from the latest file (in file order);
+  the older rows are rejected as duplicates. Within one file the first row wins. The same holds
+  for `merge_scd2`. Under `policy.severity: abort` a repeated key aborts the later file instead.
 - On key match: update non-key columns from source, only when a value changed, so rerunning a
   batch writes nothing.
 - `sink.accepted.merge.ignore_columns` can exclude additional business columns from SCD1 update sets.
@@ -41,7 +44,8 @@ Each entity's accepted rows land in one Iceberg catalog commit.
 - Strict schema handling remains the default behavior.
 - `schema.schema_evolution.mode: add_columns` applies as for standard writes; merge-key columns
   cannot be introduced by schema evolution.
-- `--full-refresh` is rejected with merge modes.
+- `--full-refresh` replaces the table with the inputs' rows in one commit (written as `overwrite`, with the
+  merge-key checks above), so rows deleted at the source leave the table. `merge_scd2` rejects it.
 - Merge-on-read leaves position delete files; `floe maintain` expires old snapshots and removes
   orphan files (see [CLI](cli.md)).
 

{floe_python-0.8.4 → floe_python-0.8.6}/orchestrators/dagster-floe/README.md

@@ -189,6 +189,13 @@ floe manifest generate -c orders.yml -p prod.yml --output manifests/orders.json
 
 See [docs/lineage.md](../../docs/lineage.md) for the full lineage config reference.
 
+## Full refresh for one run
+
+Tag a Dagster run `floe/full_refresh=true` (in the Launchpad, with `dagster job launch --tags '{"floe/full_refresh": "true"}'`,
+or in a schedule's `tags`) and every entity it materializes runs `floe run ... --full-refresh`, whatever the runner
+(in-process, `local_process`, `kubernetes_job`, `databricks_job`, either `job_scope`). Runs without the tag
+are unchanged. See [docs/cli.md](../../docs/cli.md) for what `--full-refresh` does in each write mode.
+
 ## Output metadata
 
 Every accepted and rejected asset carries `floe/entity` (the manifest entity name) and `floe/output` (`"accepted"` or `"rejected"`) in its asset metadata, so tooling can tell the two apart without parsing dagster-floe's output names: `spec.metadata["floe/output"]` on each `AssetSpec` of `multi_asset.specs`.

{floe_python-0.8.4 → floe_python-0.8.6}/orchestrators/dagster-floe/src/floe_dagster/assets.py

@@ -41,6 +41,8 @@ from .runner import RunResult, Runner
 # Advisory Dagster tag (#924): entities writing one Iceberg table share its value, so a
 # `tag_concurrency_limits` entry on this key keeps their runs apart.
 TABLE_TAG = "floe/table"
+# Run tag (#1091): `floe/full_refresh=true` runs this Dagster run's entities with `floe run --full-refresh`.
+FULL_REFRESH_TAG = "floe/full_refresh"
 
 SourceKeyFn = Callable[[ManifestEntity], AssetKey | None]
 
@@ -320,7 +322,7 @@ def _make_entity_multi_asset(
             run_id=run_id,
             entity=entity_name,
             log_format=execution.log_format,
-            execution=execution,
+            execution=_for_run(execution, run),
             runner_definition=runner_definition,
             dagster_job_name=dagster_job_name,
         )
@@ -347,6 +349,14 @@ def _make_entity_multi_asset(
     return _multi_asset
 
 
+def _for_run(execution: ManifestExecution, run: Any) -> ManifestExecution:
+    """`execution`, with `--full-refresh` appended when the Dagster run carries the full refresh tag."""
+    tags = getattr(run, "tags", None) or {}
+    if tags.get(FULL_REFRESH_TAG) != "true" or "--full-refresh" in [*execution.base_args, *execution.per_entity_args]:
+        return execution
+    return replace(execution, per_entity_args=[*execution.per_entity_args, "--full-refresh"])
+
+
 def _parse_finished(context: Any, result: RunResult, execution: ManifestExecution, run_id: str | None):
     try:
         events = parse_json_event_lines(result.stdout)
@@ -574,7 +584,9 @@ def _make_domain_multi_asset(
             run_id=floe_run_id,
             domain=domain,
             entities=names,
-            execution=replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution,
+            execution=_for_run(
+                replace(execution, per_entity_args=["--domain", domain]) if whole_domain else execution, run
+            ),
             runner_definition=runner_definition,
             dagster_run_id=run_id,
             dagster_job_name=dagster_job_name,

{floe_python-0.8.4 → floe_python-0.8.6}/orchestrators/dagster-floe/tests/test_config_mode.py

@@ -38,7 +38,7 @@ def paths(tmp_path: Path) -> tuple[str, str, str]:
     return str(config), str(profile), str(manifest)
 
 
-def _k8s_args(**source: str) -> tuple[list[str], list[list[str]]]:
+def _k8s_args(tags: dict[str, str] | None = None, **source: str) -> tuple[list[str], list[list[str]]]:
     """Asset keys, and the command each materialized entity's Kubernetes Job runs."""
     calls: list[list[str]] = []
 
@@ -49,7 +49,7 @@ def _k8s_args(**source: str) -> tuple[list[str], list[list[str]]]:
     defs = load_floe_assets(**source, runner=LocalRunner(), domains=["sales"])
     assets = list(defs.assets or [])
     with patch("floe_dagster.kubernetes_runner.run_kubernetes_job", job):
-        materialize(assets, raise_on_error=False)
+        materialize(assets, raise_on_error=False, tags=tags)
     keys = sorted(k.to_user_string() for a in assets for k in ([a.key] if isinstance(a, SourceAsset) else a.keys))
     return keys, sorted(calls)
 
@@ -99,6 +99,33 @@ def test_in_process_runner_runs_from_the_config_and_profile(paths) -> None:
     assert sorted(calls, key=str) == expected
 
 
+def test_full_refresh_run_tag_adds_the_flag_to_kubernetes_job_runs(paths) -> None:
+    _, _, manifest = paths
+    tail = ["--log-format", "json", "--quiet", "--entities"]
+    _, tagged = _k8s_args(tags={"floe/full_refresh": "true"}, manifest_path=manifest)
+    _, untagged = _k8s_args(manifest_path=manifest)
+    assert tagged == [["floe", "run", "--manifest", manifest, *tail, e, "--full-refresh"] for e in ("invoices", "orders")]
+    assert untagged == [["floe", "run", "--manifest", manifest, *tail, e] for e in ("invoices", "orders")]
+
+
+def test_full_refresh_run_tag_runs_in_process_with_full_refresh(paths) -> None:
+    config, _, _ = paths
+    calls: list[object] = []
+
+    class Outcome:
+        summary = {"run": {"exit_code": 0}}
+
+    def run(config_path: str, **kwargs: object) -> Outcome:
+        calls.append(kwargs["full_refresh"])
+        return Outcome()
+
+    defs = load_floe_assets(config_path=config, runner=InProcessRunner(), domains=["sales"])
+    with patch.object(floe, "run", run):
+        materialize(list(defs.assets or []), raise_on_error=False, tags={"floe/full_refresh": "true"})
+        materialize(list(defs.assets or []), raise_on_error=False)
+    assert calls == [True, True, False, False]
+
+
 def _domain_op_names(config: str, profile: str, **kwargs: str) -> set[str]:
     defs = load_floe_assets(config_path=config, profile_path=profile, runner=LocalRunner(), **kwargs)
     return {a.op.name for a in defs.assets or [] if not isinstance(a, SourceAsset) and a.op.name.startswith("floe_")}

{floe_python-0.8.4 → floe_python-0.8.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "floe-python"
-version = "0.8.4"
+version = "0.8.6"
 description = "Floe: a data-contract runtime for file ingestion"
 requires-python = ">=3.11"
 dependencies = [

{floe_python-0.8.4 → floe_python-0.8.6}/src/floe/checks/normalize.py

@@ -94,12 +94,12 @@ def output_column_mapping(
     targets: dict[str, str] = {}
     mapping: dict[str, str] = {}
     for column, target in zip(columns, output_names(columns, strategy), strict=True):
-        source = normalize_name(column.source or column.name, strategy)
         if target in targets:
             raise FloeConfigError(
-                f"output column name collision: {targets[target]} and {source} -> {target}"
+                f"output column name collision: {targets[target]} and {column.name} -> {target}"
             )
-        targets[target] = source
+        targets[target] = column.name
+        source = normalize_name(column.source or column.name, strategy)
         if source != target:
             mapping[source] = target
     return mapping

{floe_python-0.8.4 → floe_python-0.8.6}/src/floe/config/validate.py

@@ -6,6 +6,7 @@ from floe import __version__
 from floe.checks.cast import decimal_type
 from floe.checks.normalize import (
     normalize_name,
+    output_column_mapping,
     output_names,
     resolve_strategy,
     source_names,
@@ -416,7 +417,9 @@ def validate_schema(entity: EntityConfig, version: str) -> None:
             f"(allowed: {', '.join(NORMALIZE_STRATEGIES)})",
         )
     try:
-        source_names(schema.columns, resolve_strategy(schema))
+        resolved = resolve_strategy(schema)
+        source_names(schema.columns, resolved)
+        output_column_mapping(schema.columns, resolved)
     except FloeConfigError as err:
         raise _fail(entity, str(err)) from err
     mismatch = schema.mismatch

{floe_python-0.8.4 → floe_python-0.8.6}/src/floe/iceberg/sink.py

@@ -177,7 +177,8 @@ def write_iceberg(
     key = [output.get(k.strip(), k.strip()) for k in primary_key]
     identifier = (target.namespace, target.table)
     with run_error("rest catalog table_exists check failed"):
-        exists = mode in ("merge_scd1", "merge_scd2") and catalog.table_exists(identifier)  # fmt: skip
+        # the configured mode: a refreshed merge writes as overwrite and keeps the key guard (#1091)
+        exists = entity.sink.write_mode in ("merge_scd1", "merge_scd2") and catalog.table_exists(identifier)  # fmt: skip
     if exists:
         with run_error("rest catalog load_table failed"):
             existing = set(catalog.load_table(identifier).schema().column_names)

{floe_python-0.8.4 → floe_python-0.8.6}/src/floe/run/__init__.py

@@ -72,10 +72,10 @@ def _rfc3339() -> str:
 
 
 def _refresh(entity: EntityConfig) -> EntityConfig:
-    """Full refresh: an append sink overwrites; merge sinks cannot refresh."""
+    """Full refresh: an append sink overwrites; `merge_scd1` overwrites in `run_entity`, which keeps its merge checks."""
     mode = entity.sink.write_mode
-    if mode in ("merge_scd1", "merge_scd2"):
-        raise FloeConfigError(f"entity '{entity.name}': --full-refresh is not supported with write_mode '{mode}'")  # fmt: skip
+    if mode == "merge_scd2":
+        raise FloeConfigError(f"entity '{entity.name}': --full-refresh is not supported with write_mode '{mode}': its version history cannot be rebuilt from the inputs")  # fmt: skip
     if mode != "append":
         return entity
     return entity.model_copy(update={"sink": entity.sink.model_copy(update={"write_mode": "overwrite"})})  # fmt: skip

{floe_python-0.8.4 → floe_python-0.8.6}/src/floe/run/entity.py

@@ -848,6 +848,11 @@ def _run_entity(
     entity = with_partition_columns(
         entity
     )  # D28: the derived columns are schema columns
+    merging = entity.sink.write_mode in ("merge_scd1", "merge_scd2")
+    configured = entity  # the sink still checks a merge's key against the table
+    # #1091: a full refresh rebuilds a merge_scd1 table from every input in one commit
+    if inc.full_refresh and entity.sink.write_mode == "merge_scd1":
+        entity = entity.model_copy(update={"sink": entity.sink.model_copy(update={"write_mode": "overwrite"})})  # fmt: skip
     sink = entity.sink
     if sink.accepted.format not in ("parquet", "iceberg"):
         raise FloeRunError(f"entity.name={entity.name} unsupported sink format: {sink.accepted.format}")  # fmt: skip
@@ -885,7 +890,7 @@ def _run_entity(
     if location and entity.incremental_mode in ("none", "archive"):
         plan.archive = location[0], storage_client(location[1])
         plan.root = source_root(source_uri) if entity.source.partitioning else None
-    if sink.write_mode in ("merge_scd1", "merge_scd2") and entity.schema_.primary_key:
+    if merging and entity.schema_.primary_key:
         pk = [runtime.get(k.strip(), k.strip()) for k in entity.schema_.primary_key]
         plan.merge_key = (
             pk[0] if len(pk) == 1 else f"[{','.join(pk)}]"
@@ -928,6 +933,9 @@ def _run_entity(
     skipped = len(files)
     downloader = None if isinstance(client, LocalClient) else client
     failure: list[Exception] = []
+    # #1090: a merge keeps a key's latest version, so the newest file is checked first; the first row of a file still wins.
+    # Not under abort: a duplicate aborts there anyway, and newer files committed ahead of a pending older one would let it merge stale rows later
+    latest_first = merging and entity.policy.severity != "abort"
 
     def accepted_frames() -> Generator[pl.DataFrame]:
         """Files are read as the sink pulls (#1056); `files` and `complete` are final once it is exhausted."""
@@ -936,7 +944,7 @@ def _run_entity(
             with tempfile.TemporaryDirectory(prefix="floe-") as temp_dir:
                 if pending and sink.write_mode == "append" and plan.tracker.keys and (target is None or plan.table is not None):  # fmt: skip
                     plan.history = _key_cache(plan, accepted_client, base, Path(temp_dir))  # fmt: skip
-                for f in pending:
+                for f in pending[::-1] if latest_first else pending:
                     emit("file_started", entity=entity.name, input=f.source_uri)
                     timer = time.monotonic()
                     local = localize_input(f, downloader, Path(temp_dir))
@@ -973,7 +981,7 @@ def _run_entity(
                 write = write_accepted(frames, accepted_client, base, sink.write_mode, sink.accepted.options)  # fmt: skip
             else:
                 outputs = [plan.out_map.get(n, n) for n in names]
-                result, table, merge, added = write_iceberg(entity, target, frames, outputs, sink.write_mode, emit.run_id, lambda: input_hash(entity, complete))  # fmt: skip
+                result, table, merge, added = write_iceberg(configured, target, frames, outputs, sink.write_mode, emit.run_id, lambda: input_hash(entity, complete))  # fmt: skip
                 write = AcceptedWrite(result.files_written, result.file_paths, result.total_bytes_written,
                                       result.avg_file_size_mb, result.small_files_count)  # fmt: skip
                 if added:
@@ -989,11 +997,15 @@ def _run_entity(
         raise failure[0] from failure[0].__cause__
     finally:
         stream.close()
+    if latest_first:  # the report lists the files in file order
+        files[skipped:] = files[skipped:][::-1]
     files[skipped:] = sorted(files[skipped:], key=lambda r: r["input_file"] not in plan.prechecked)  # fmt: skip
     inc.committed = files  # #915: from here on a failure promotes the written files
     # #914: archive only once the sink committed, so a failed commit leaves the inputs in the source (Rust archives before)
     by_uri = {r["input_file"]: r for r in files}
-    for f in plan.to_archive:
+    order = {f.source_uri: i for i, f in enumerate(inputs.files)}
+    # #1090: oldest first, so a partial archive leaves the newest batch pending, never only an older one
+    for f in sorted(plan.to_archive, key=lambda f: order[f.source_uri]) if latest_first else plan.to_archive:  # fmt: skip
         by_uri[f.source_uri]["output"]["archived_path"] = _archive(plan, f)
     unique = [_report_names(r, key, runtime) for r, key in zip(plan.tracker.results(), keys, strict=True)]  # fmt: skip
     report = build_report(

{floe_python-0.8.4 → floe_python-0.8.6}/tests/integration/test_iceberg_merge.py

@@ -39,6 +39,7 @@ from pyiceberg.types import (
 from floe.config.storage import CatalogDefinition, StorageDefinition
 from floe.iceberg.catalog import ensure_table, open_catalog, open_duckdb
 from floe.iceberg.merge import merge_scd1, merge_scd2
+from floe.iceberg.write import write_table
 
 URI = os.environ.get("FLOE_POLARIS_URI")
 
@@ -212,6 +213,21 @@ def test_ignored_columns_keep_target_values(catalog: Catalog, table: Table) -> N
     assert read(table)[0] == (1, "a2", 10.0, D(2026, 1, 1), "eu")
 
 
+def test_a_full_refresh_replaces_a_merged_table(catalog: Catalog, table: Table) -> None:
+    """#1091: a `merge_scd1` full refresh writes as `overwrite` over a merge-on-read table, in one commit."""
+    con = open_duckdb(CATALOG, STORAGE)
+    merge_scd1(con, table, arrow(ARROW, [(1, "a2", 11.0, D(2026, 1, 1), "eu")]).to_reader(), ["id"])  # fmt: skip
+    commits = len(table.metadata.metadata_log)
+    inputs = [(1, "a3", 12.0, D(2026, 1, 1), "eu"), (9, "z", None, D(2026, 1, 9), "us")]
+
+    write_table(table, arrow(ARROW, inputs).to_reader(), "overwrite", "refresh", "hash")
+
+    # rows gone from the inputs leave the table, with their deletes
+    assert read(table) == inputs
+    assert len(table.metadata.metadata_log) == commits + 1
+    assert_files_hold_their_partition(table)
+
+
 # SCD2: #598's golden fixture; the target also holds a closed historical version of key 1
 SYSTEM = [
     NestedField(6, "__floe_is_current", BooleanType(), required=True),

{floe_python-0.8.4 → floe_python-0.8.6}/tests/unit/config/test_validate.py

@@ -194,6 +194,17 @@ def test_columns_that_normalise_to_one_name_fail_validation(
     assert str(err.value) == f"entity.name=orders column source collision: {message}"
 
 
+def test_output_column_name_collision_fails_validation(tmp_path: Path) -> None:
+    columns = "[{name: a, source: X, type: string}, {name: A, type: string}]"
+    schema = f"{{columns: {columns}, normalize_columns: {{enabled: true}}}}"
+    with pytest.raises(FloeConfigError) as err:
+        check(tmp_path, schema)
+    assert (
+        str(err.value)
+        == "entity.name=orders output column name collision: a and A -> a"
+    )
+
+
 def test_valid_spellings(tmp_path: Path) -> None:
     check(
         tmp_path,

{floe_python-0.8.4 → floe_python-0.8.6}/tests/unit/run/test_run_iceberg.py

@@ -412,6 +412,46 @@ def test_merge_scd1_upserts_and_reports_the_counts(tmp_path: Path, catalog: Loca
     assert out["merge_elapsed_ms"] >= 0 and out["snapshot_id"] is not None
 
 
+def test_merge_keeps_the_key_of_the_latest_pending_file(tmp_path: Path, catalog: LocalCatalog, merges: Merges) -> None:  # fmt: skip
+    """#1090: a key repeated across pending files resolves to the latest file; within a file the first row wins."""
+    (tmp_path / "in").mkdir()
+    (tmp_path / "in" / "a.csv").write_text("id,name\n1,old\n2,b\n")
+    (tmp_path / "in" / "b.csv").write_text("id,name\n1,new\n1,newer\n")
+    root = config(tmp_path, mode="merge_scd1")
+    report = run_entity(root.entities[0], tmp_path, "0.3", storages=root.storages, catalogs=root.catalogs).report  # fmt: skip
+    assert rows(catalog) == [(1, "new"), (2, "b")]
+    files = [(Path(f["input_file"]).name, f["accepted_count"], f["rejected_count"]) for f in report["files"]]  # fmt: skip
+    assert files == [("a.csv", 1, 1), ("b.csv", 1, 1)]  # reported in file order
+
+
+def test_merge_archives_the_inputs_oldest_first(tmp_path: Path, catalog: LocalCatalog, merges: Merges, monkeypatch: pytest.MonkeyPatch) -> None:  # fmt: skip
+    """#1090: a partial archive must leave the newest batch pending, never only an older one a retry would merge over it."""
+    (tmp_path / "in").mkdir()
+    (tmp_path / "in" / "a.csv").write_text("id,name\n1,old\n")
+    (tmp_path / "in" / "b.csv").write_text("id,name\n1,new\n")
+    config(tmp_path, mode="merge_scd1")
+    path = tmp_path / "config.yml"
+    path.write_text(path.read_text().replace("      rejected:", "      archive: {path: ./arch}\n      rejected:"))  # fmt: skip
+    root = parse_config(path)
+    archived: list[str] = []
+    archive = entity_module._archive  # pyright: ignore[reportPrivateUsage] -- wrapped to record the order
+    monkeypatch.setattr(entity_module, "_archive", lambda plan, f: archived.append(f.source_name) or archive(plan, f))  # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType] -- recorder  # fmt: skip
+    run_entity(root.entities[0], tmp_path, "0.3", storages=root.storages, catalogs=root.catalogs)  # fmt: skip
+    assert rows(catalog) == [(1, "new")]
+    assert archived == ["a.csv", "b.csv"]
+
+
+def test_merge_under_abort_keeps_the_file_order(tmp_path: Path, catalog: LocalCatalog, merges: Merges) -> None:  # fmt: skip
+    """#1090: abort commits the files before the aborted one, so the older file goes first and a retry never merges it over a newer one."""
+    (tmp_path / "in").mkdir()
+    (tmp_path / "in" / "a.csv").write_text("id,name\n1,old\n")
+    (tmp_path / "in" / "b.csv").write_text("id,name\n1,new\n")
+    root = config(tmp_path, mode="merge_scd1", severity="abort")
+    report = run_entity(root.entities[0], tmp_path, "0.3", storages=root.storages, catalogs=root.catalogs).report  # fmt: skip
+    assert [(Path(f["input_file"]).name, f["status"]) for f in report["files"]] == [("a.csv", "success"), ("b.csv", "aborted")]  # fmt: skip
+    assert rows(catalog) == [(1, "old")]
+
+
 def test_merge_scd2_closes_the_changed_row_and_reports_the_counts(
     tmp_path: Path,
     server: tuple[RestCatalog, CatalogDefinition],  # noqa: F811 -- the imported fixture
@@ -439,15 +479,26 @@ def test_merge_scd2_closes_the_changed_row_and_reports_the_counts(
     assert out["merge_elapsed_ms"] >= 0 and out["snapshot_id"] is not None
 
 
-def test_full_refresh_with_a_merge_is_a_config_error(tmp_path: Path, catalog: LocalCatalog) -> None:  # fmt: skip
+def test_full_refresh_with_merge_scd2_is_a_config_error(tmp_path: Path, catalog: LocalCatalog) -> None:  # fmt: skip
     (tmp_path / "in").mkdir()
     (tmp_path / "in" / "orders.csv").write_text("id,name\n1,a\n")
-    root = config(tmp_path, mode="merge_scd1")
-    with pytest.raises(FloeConfigError, match="^entity 'Orders': --full-refresh is not supported with write_mode 'merge_scd1'$"):  # fmt: skip
+    root = config(tmp_path, mode="merge_scd2")
+    with pytest.raises(FloeConfigError, match="^entity 'Orders': --full-refresh is not supported with write_mode 'merge_scd2': its version history cannot be rebuilt from the inputs$"):  # fmt: skip
         run_project(tmp_path / "config.yml", root, full_refresh=True)
     assert not catalog.tables
 
 
+def test_full_refresh_with_merge_scd1_rebuilds_the_table_from_the_inputs(tmp_path: Path, catalog: LocalCatalog, merges: Merges) -> None:  # fmt: skip
+    """#1091: a row gone from the inputs leaves the table; a key in two files keeps the latest (#1090)."""
+    run(tmp_path, "id,name\n1,a\n2,b\n9,gone\n", mode="merge_scd1")
+    (tmp_path / "in" / "orders.csv").write_text("id,name\n1,old\n2,b\n")
+    (tmp_path / "in" / "zz.csv").write_text("id,name\n1,new\n")
+    outcome = run_project(tmp_path / "config.yml", config(tmp_path, mode="merge_scd1"), full_refresh=True)  # fmt: skip
+    assert rows(catalog) == [(1, "new"), (2, "b")]
+    assert len(merges) == 1  # the refresh replaced the table instead of merging
+    assert outcome.summary["results"]["rejected_total"] == 1
+
+
 def test_input_hash_covers_the_entity_and_the_claimed_files(tmp_path: Path) -> None:  # fmt: skip
     entity = config(tmp_path).entities[0]
     a = InputFile("local:///in/a.csv", "a.csv", "a", 1, "t1")
@@ -483,6 +534,10 @@ def test_a_merge_key_column_cannot_be_added_by_evolution(tmp_path: Path, catalog
     with pytest.raises(FloeRunError, match="iceberg schema evolution failed: merge key columns cannot be added: qty$"):  # fmt: skip
         run_entity(root.entities[0], tmp_path, "0.3", storages=root.storages, emit=NO_EVENTS, catalogs=root.catalogs)  # fmt: skip
     assert rows(catalog) == [(1, "a")]
+    # #1091: a refresh keeps the guard
+    with pytest.raises(FloeRunError, match="merge key columns cannot be added: qty$"):
+        run_project(tmp_path / "config.yml", root, full_refresh=True)
+    assert rows(catalog) == [(1, "a")]
 
 
 def test_d28_partition_spec_on_the_derived_date(tmp_path: Path, catalog: LocalCatalog) -> None:  # fmt: skip

{floe_python-0.8.4 → floe_python-0.8.6}/uv.lock

@@ -450,7 +450,7 @@ wheels = [
 
 [[package]]
 name = "floe-python"
-version = "0.8.4"
+version = "0.8.6"
 source = { editable = "." }
 dependencies = [
     { name = "boto3" },