iceberg-ops 0.1.0__tar.gz

iceberg_ops-0.1.0/.gitignore

@@ -0,0 +1,70 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+.env
+
+# Testing / coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.nox/
+.cache/
+coverage.xml
+*.cover
+.hypothesis/
+
+# Type / lint caches
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+.ruff_cache/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Sphinx
+docs/_build/
+
+# Jupyter
+.ipynb_checkpoints/
+
+# iceberg-ops runtime data
+local_data/

iceberg_ops-0.1.0/CHANGELOG.md

@@ -0,0 +1,502 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-11-08
+
+First public release of `iceberg-ops`: Alembic-style schema migrations
+for Apache Iceberg, plus a cross-table data mover, basic maintenance
+helpers and an experimental Apache Polaris policy executor.
+
+Development builds from May 2026 were never published. The Changed and
+Fixed lists below describe how this release differs from them, for
+anyone who ran those builds from source.
+
+### Added
+
+Schema migrations:
+
+- `BaseMigration` and `MigrationContext` for `migrations/NNNN_*.py`
+  files with `upgrade()` / `downgrade()` and the `pre_upgrade`,
+  `post_upgrade`, `pre_downgrade` and `post_downgrade` hooks. Context
+  helpers: `create_table_if_not_exists` (with `partition_spec`,
+  `sort_order`, `properties` and `location`, applied on create),
+  `drop_table_if_exists`, `load_table`, `add_column`, `enforce_column`
+  (with opt-in `legacy_archival`), `enforce_rename`, `delete_column`,
+  `update_partition_spec`, `replace_sort_order`, `set_clustering_keys`,
+  `set_table_properties`, `unset_table_properties`,
+  `set_snapshot_retention` and `set_compaction_target_size`. Helpers
+  take bare table names; the runner adds the namespace.
+- `MigrationRunner` with `migrate`, `rollback`, `stamp`, `plan`,
+  `status` and `history` for one namespace at a time. Migration history
+  is an Iceberg table, `_iceberg_ops.migration_history`, in the managed
+  catalog; it keeps the latest status of each version per namespace.
+- `iceberg_ops.runner.normalize_version`: `"2"`, `"0002"` and `2` name
+  the same version.
+- The `iceberg-ops` CLI: `init`, `revision`, `migrate`, `rollback`,
+  `stamp`, `plan`, `status`, `current`, `history`, `migrate-data` and
+  `run-policies`, plus `--version`. Settings come from flags or from
+  `iceberg-ops.yaml` (keys `catalog`, `catalog_name`, `namespace`,
+  `db_path`, `warehouse`, `migrations_dir`).
+- `--catalog-name NAME` and the config key `catalog_name` open any
+  catalog defined in `.pyiceberg.yaml` or `PYICEBERG_CATALOG__<NAME>__*`
+  environment variables. `migrate-data` has `--src-catalog-name` /
+  `--dst-catalog-name` and `run-policies` has `--catalog-name`.
+- Autogenerate (`iceberg_ops.autogen`): `schema_diff_to_ops`,
+  `spec_diff_to_ops`, `sort_order_diff_to_ops`, `properties_diff_to_ops`,
+  `render_migration_body`, `render_migration_file` and
+  `write_autogenerated_revision`, plus `iceberg-ops revision
+  --autogenerate --target-table T` with `--target-schema-yaml FILE` or
+  `--target-schema-arrow module:attr` and repeatable
+  `--rename-hint old:new`. The CLI diffs columns; the Python API also
+  diffs partition spec, sort order and table properties. Rename hints
+  are validated, and new columns keep their target doc.
+- YAML schema contracts (`SchemaContract.from_yaml`) accept the Iceberg
+  primitive types, including `decimal(P,S)`, `fixed[N]` and
+  `timestamp_ns`, and are validated with clear errors.
+- Core helpers (`iceberg_ops.core`): `CatalogFactory.build` for the
+  `sqlite`, `memory`, `glue`, `rest`, `hive` and `s3tables` kinds,
+  `default_local_paths()`, Arrow-to-Iceberg type translation with
+  `is_allowed_promotion(old, new)` (Arrow time, uuid, fixed-size binary,
+  string_view, binary_view and dictionary types are mapped), schema
+  diff, `SchemaContract`, `SchemaEvolutionHelper`, and snapshot, upsert,
+  time-travel and field-history utilities.
+
+Companion modules:
+
+- Cross-table data migration, experimental (`iceberg_ops.data_migration`,
+  `iceberg-ops migrate-data`): `SchemaReconciler`, `CheckpointStore` and
+  `DataMigrator` with `append`, `upsert` and `overwrite_partition` write
+  modes, checkpoint resume, dry runs and `row_count` /
+  `partition_hash` / `sample` verification. `MigrationResult` reports
+  `dest_snapshot_before` (a rollback target), `rows_inserted` and
+  `rows_updated`; `CheckpointStore.job` exposes the recorded job
+  settings. Every snapshot the migrator writes carries
+  `iceberg-ops.migration-job` and `iceberg-ops.migration-batch` summary
+  properties.
+- Basic maintenance helpers for self-managed tables
+  (`iceberg_ops.maintenance`): `compact_small_files_streaming`,
+  `expire_snapshots`, `remove_orphan_files` and `table_health_report`.
+  `SnapshotExpiryResult.protected_ids` lists snapshots kept because a
+  branch or tag points at them; `OrphanCleanupResult.failed_paths` lists
+  orphans whose deletion failed.
+- Apache Polaris policy executor, experimental (`iceberg_ops.polaris`,
+  `iceberg-ops run-policies`): `PolarisClient`, `Policy`,
+  `parse_policy`, `fetch_table_policies`, `dispatch_policy`,
+  `dispatch_policies` and a `handlers={...}` hook for custom policy
+  types. Only `system.snapshot-expiry` is executed.
+- `iceberg_ops.testing`: `FakeResponse` and `FakeSession` test doubles
+  for driving the Polaris client without HTTP. They may change before
+  1.0.
+
+Packaging and project:
+
+- Extras: `sqlite`, `glue`, `hive`, `s3tables`, `transforms` (needed to
+  write to tables partitioned by bucket, truncate or time transforms)
+  and `polaris`. `rest` and `contracts` are accepted but add nothing to
+  the core install.
+- A `py.typed` marker, so type checkers use the package's annotations.
+- Documentation at https://namitkewat.github.io/iceberg-ops/, including
+  a compatibility page and plain-text `llms.txt` / `llms-full.txt`
+  exports of the docs.
+- `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` (Contributor Covenant 3.0),
+  `SECURITY.md` (GitHub private vulnerability reporting), `ROADMAP.md`,
+  issue forms and a pull request template.
+- `AGENTS.md`, a guide for coding agents working on the repository, and
+  `agent-skills/iceberg-ops-migrations/SKILL.md`, a skill that teaches a
+  coding agent to write, review and run iceberg-ops migrations.
+
+### Changed
+
+Packaging and CLI:
+
+- The command-line tool is `iceberg-ops` only, and its config file is
+  `iceberg-ops.yaml`. The earlier short script name and its config file
+  name were dropped: that name belongs to an unrelated project.
+- PyIceberg is pinned to `>=0.12.0,<0.13` (was `>=0.11.0,<0.12`) and
+  PyArrow to `>=18`. Python 3.14 is tested in CI. PyYAML is a core
+  dependency, and `tzdata` is installed on Windows.
+- The `[maintenance]` extra was removed: it only installed boto3, which
+  no module imported. `[s3tables]` installs `pyiceberg[rest-sigv4]`;
+  `[rest]` and `[polaris]` no longer request a `pyiceberg[rest]` extra,
+  which PyIceberg does not define.
+- Development status is Alpha.
+- Default local state: without `db_path` / `warehouse_dir`,
+  `CatalogFactory.build("sqlite")` uses `./local_data/` in the current
+  working directory at call time, and the CLI's sqlite catalog uses
+  `local_data/` next to the config file (or in the current directory
+  when there is none). `CatalogFactory.build("memory")` without
+  `warehouse_dir` writes data files to a fresh temporary directory.
+- The `glue`, `rest`, `hive` and `s3tables` kinds are loaded through
+  `pyiceberg.catalog.load_catalog`, so `.pyiceberg.yaml` and
+  `PYICEBERG_CATALOG__<KIND>__*` variables apply; explicit arguments
+  win. `db_path` with a remote kind raises `ValueError`, and the local
+  kinds reject `uri` / `warehouse` keyword arguments.
+- The CLI no longer accepts the memory catalog kind, which forgot
+  applied migrations after each command. Catalog-kind options are
+  case-insensitive, and a typo is a usage error.
+- Relative paths in the config file (`db_path`, the sqlite `warehouse`,
+  `migrations_dir`) resolve against the config file's directory, and
+  `migrations/` defaults to sitting next to it. Paths given as flags
+  stay relative to the current directory.
+- CLI exit codes: 2 for usage errors (bad flags or config, missing
+  catalog, namespace or migrations directory, invalid `--target` /
+  `--steps`); 1 for runtime failures, with a single `Error:` line. Set
+  `ICEBERG_OPS_TRACEBACK=1` to see the traceback.
+- A missing migrations directory is an error (`FileNotFoundError` from
+  the runner, a usage error in the CLI) instead of reporting that all
+  migrations are up to date. An explicit `--config` path that does not
+  exist is a usage error, except for `init`.
+
+Migrations:
+
+- A migration's `version` must equal its file's `NNNN` prefix, and two
+  files with the same version are rejected. `revision` numbers new files
+  by the same `NNNN_<description>.py` rule the runner uses.
+- `plan`, `status` and `history` no longer create the `_iceberg_ops`
+  namespace or the history table; the first write creates them.
+- `--target` / `target=` is compared as a version number: `2` means
+  `0002`, and an unknown or malformed target raises `ValueError` (CLI:
+  exit 2) before anything is applied. `rollback(target="0000")` rolls
+  back every migration. `rollback --steps` must be 1 or more, defaults
+  to 1 and cannot be combined with `--target`.
+- `enforce_column` adds a missing column, does nothing for an equivalent
+  type (only a changed doc is written), updates a promotable primitive
+  type in place, and otherwise raises
+  `pyiceberg.exceptions.ValidationError` unless `legacy_archival=True`.
+  Promotions follow Iceberg's rules, so string-to-binary and decimal
+  scale changes are rejected.
+- `set_clustering_keys` sets only the Iceberg sort order, in one commit.
+  It no longer writes `snowflake.cluster-by` or `delta.clustering` table
+  properties.
+- `set_snapshot_retention` logs a warning when the catalog is AWS S3
+  Tables, which manages snapshot retention itself.
+- `ConcurrentMigrationError` is defined in `iceberg_ops.history` and is
+  still importable from `iceberg_ops.runner`.
+- `revision` never overwrites an existing migration file (exit 1).
+  `write_autogenerated_revision` refuses to overwrite one unless
+  `overwrite=True`, and always writes LF line endings.
+
+Autogenerate:
+
+- A bare `--target-table` stays bare in the generated file, so the
+  migration applies to whichever `--namespace` it runs against. A dotted
+  name pins the file to that table and prints a warning.
+- A generated `upgrade()` or `downgrade()` raises `NotImplementedError`
+  before any step runs when one of its steps cannot run or cannot be
+  undone (a column drop, a type widening, a partition-field removal, a
+  sort-order replacement, a property removal with an unknown old
+  value). The generated calls are kept as comments.
+- Type-change downgrades are no longer generated as narrowing calls,
+  since Iceberg only widens types; they raise `NotImplementedError`.
+- A non-strict `SchemaContract` (the default) keeps the table columns
+  it does not list, and the CLI reports them. Set `strict: true` to drop
+  them. Nullability and doc differences on existing columns are
+  reported, not emitted.
+
+Data migration and maintenance:
+
+- `write_mode="overwrite_partition"` needs a destination partitioned by
+  identity transforms only, and the migration must fit in one commit (at
+  most `batch_rows` source rows after the row filter). Anything else is
+  rejected before any write.
+- Checkpoints record the job's settings (source and destination table
+  UUIDs, row filter, `batch_rows`, write mode, upsert keys and
+  reconciler). Resuming with different settings, a recreated
+  destination or a checkpoint without these settings raises
+  `CheckpointError`, which is now a `RuntimeError`. A completed
+  checkpoint is terminal: call `CheckpointStore.reset()` to run again.
+- `MigrationResult.new_snapshots` lists every destination snapshot the
+  run created (up to three per upsert batch, or per 1,000-key sub-batch
+  for composite keys), and
+  `CheckpointStore.mark_committed` accepts a list of snapshot IDs.
+- `DataMigrator` rejects blank or repeated `upsert_keys`, `upsert_keys`
+  with a write mode other than upsert, and upsert keys missing from the
+  projected destination schema. An upsert with
+  `on_missing_in_source="skip"` that would drop destination columns is
+  rejected; with `"null"`, the plan warns that matched keys get those
+  values overwritten with null. `SchemaReconciler.compute_plan` reports
+  a `column_mapping` key that is not a source column.
+- `migrate-data --batch-rows` and `--sample-size` must be at least 1,
+  `--upsert-keys` ignores empty entries, and a missing destination table
+  explains that `migrate-data` does not create it.
+- `remove_orphan_files` defaults to `dry_run=True`; pass
+  `dry_run=False` to delete. It refuses tables with `gc.enabled=false`
+  (in a dry run too), rejects a negative `older_than_hours` and `sweep`
+  entries that are absolute or escape the table location, and warns when
+  `older_than_hours` is below 24. `deleted` is True only when at least
+  one file was removed.
+- All `expire_snapshots` arguments after `table` are keyword-only, and
+  `max_age_hours` accepts fractional hours. Omitted arguments fall back
+  to the table's `history.expire.max-snapshot-age-ms` /
+  `history.expire.min-snapshots-to-keep` properties (as written by
+  `set_snapshot_retention`), then to the Iceberg defaults of 5 days and
+  1 snapshot. Invalid values raise `ValueError`. The fallback to
+  PyIceberg's internal `ExpireSnapshots` class for releases older than
+  0.11 was removed.
+- `compact_small_files_streaming` lost its unused `batch_rows`
+  parameter.
+
+Polaris (experimental):
+
+- Only `system.snapshot-expiry` is executed, and only with
+  `"enable": true` (a JSON boolean). `system.metadata-compaction` no
+  longer triggers a data-file rewrite; it, `system.data-compaction` and
+  `system.orphan-file-removal` are reported as skipped.
+- `system.snapshot-expiry` reads the Polaris schema
+  (`config.max_snapshot_age_days`, `config.min_snapshot_to_keep`;
+  `max_ref_age_days` is ignored with a note). Legacy top-level `*_ms`
+  keys, unknown top-level keys, unknown schema versions and invalid
+  values fail the dispatch instead of running on defaults.
+- A custom handler returning `None` yields empty details; any other
+  non-dict return marks the dispatch as failed.
+- `run-policies` is marked `[experimental]` and prints a warning when
+  run, because its requests do not yet match the released Polaris
+  policy API.
+
+### Fixed
+
+Migrations and CLI:
+
+- Two concurrent `migrate` runs for the same namespace no longer leave
+  duplicate history rows on PyIceberg 0.12
+  (apache/iceberg-python#4194). The run that records second raises
+  `ConcurrentMigrationError`, and history keeps one row per version and
+  namespace. Existing duplicate rows are read as their latest status and
+  removed by the next write for that version.
+- When a migration's own commit conflicts with a concurrent commit, it
+  raises `ConcurrentMigrationError` and records nothing, instead of
+  overwriting the other run's history with `failed`.
+- `migrate --target 2` no longer applies every migration, `stamp
+  --target 2` no longer stamps all of them, `rollback --target 1` is no
+  longer a no-op, and `"2"` no longer also selects `0010`.
+- `rollback(steps=0)` and a negative `steps` raise `ValueError` instead
+  of rolling back an unexpected number of migrations.
+- `rollback` checks that every migration file it needs exists before
+  running any `downgrade()`. `status` lists history entries whose
+  migration file is missing.
+- `enforce_column` is idempotent for struct, list and map columns.
+- `enforce_column(..., legacy_archival=True)` archives only on a real
+  type incompatibility. Commit errors and failing contracts propagate
+  instead of archiving the column and adding an empty replacement.
+- Archival, in `enforce_column` and in the `enforce_rename` collision
+  path, refuses columns used by the partition spec, sort order or
+  identifier fields. An archived required column becomes optional, so
+  later appends no longer fail. If archival or the rename fails part
+  way, the column is restored with its original doc and required flag;
+  if that restore also fails, a `RuntimeError` is raised.
+- `enforce_rename(name, name)` is a no-op, and renaming across
+  different parent structs raises `ValueError`.
+- `enforce_column` and `add_column` accept primitive PyArrow types such
+  as `pa.float64()` and `pa.string()`.
+- Migration files that define a `@dataclass` (with `from __future__
+  import annotations`) load. Same-named files in different namespace
+  directories no longer collide. A broken migration file raises an error
+  that names the file, and the CLI prints it as one `Error:` line.
+- `create_table_if_not_exists` works for nested namespaces such as
+  `sales.eu`, and returns the existing table if another process creates
+  it first.
+- `set_table_properties` accepts a property key named `properties`.
+- `revision --autogenerate` honours `--catalog-name`, the config's
+  `catalog_name` and the config-relative sqlite defaults.
+- Missing extras, missing tables, missing catalog properties,
+  connection failures, permission or server errors, commit conflicts and
+  corrupt checkpoints print a single `Error:` line and exit 1 instead of
+  a traceback.
+- Config file problems (invalid YAML, a non-mapping top level,
+  non-string values such as `namespace: 123`) are usage errors. Unknown
+  keys print a warning.
+- Config and generated files are read and written as UTF-8 (a BOM is
+  tolerated). `init` appends its block to `.gitignore` without rewriting
+  the existing content, and honours `--config` and `--migrations-dir`.
+- Using the `s3tables` kind without boto3 says to install
+  `iceberg-ops[s3tables]`.
+- `CatalogFactory.build(..., reset=True)` refuses to delete a
+  filesystem root, the home directory, the working directory or one of
+  its parents, or a database file that is not SQLite.
+- The `sqlite` and `memory` kinds reject a warehouse that looks like a
+  URI and local paths containing `#` or `?`, which PyIceberg would
+  misread, before anything is written.
+- The history table is created with `commit.retry.num-retries=20`, so
+  runners for different namespaces sharing it no longer lose history
+  writes to commit contention.
+- When recording a failure in history itself fails, the migration's
+  own error is still the one raised; the history error is logged.
+- `history` sizes its columns to their contents, so full ISO-8601
+  timestamps no longer push DESCRIPTION out of line, and a multi-line
+  error is printed on one line.
+- `create_table_if_not_exists` raises `ValueError` before creating
+  anything when a `pa.Schema` is combined with a non-empty
+  `partition_spec` or `sort_order`.
+- `SchemaContract.validate` compares struct, list and map types without
+  regard to field IDs, and `enforce_rename` checks the contract when
+  neither column exists.
+
+Autogenerate:
+
+- `revision --autogenerate` passes the table's partition spec, sort
+  order and format version to the diff. Dropping a column that the
+  spec, the sort order or the identifier fields use makes the generated
+  `upgrade()` raise before any step runs, and a type the table's format
+  version cannot hold is a usage error. Column names containing `.`,
+  decimals with a scale outside `0..precision` or more than 38 digits,
+  and `fixed[0]` are rejected too. Existing nested columns of a
+  `--target-schema-arrow` target that were not compared are named in a
+  warning instead of reporting no differences.
+- Partition and sort fields sourced from a nested column render as the
+  full dotted path and bind to the nested field.
+- Autogenerate no longer emits type changes that Iceberg does not
+  allow, such as a decimal scale change, string to long, or binary to
+  string; such a step makes `upgrade()` raise before anything runs.
+- A generated `downgrade()` with an irreversible step no longer rolls
+  back part of the migration and then fails.
+- Nested type changes (struct, list, map) in a `pyiceberg.schema.Schema`
+  target (Python API) produce a file that compiles, with the step
+  blocked for manual completion, instead of a syntax error.
+- Invalid YAML contracts, unknown types and bad rename hints make
+  `revision --autogenerate` exit with a usage error.
+- Partition fields with non-identity transforms (day, bucket, ...)
+  import their transform and run. Sort-order diffs compare transform
+  and null order, and reject orders the generated call cannot express.
+- `--target-schema-arrow` modules are importable from the current
+  directory when the CLI runs from the installed console script.
+
+Data migration:
+
+- `overwrite_partition` could delete rows written by an earlier batch
+  of the same migration; it is now limited to one commit (see Changed).
+- A resume reads the source snapshot recorded in the checkpoint, so rows
+  added to the source after a crash are neither skipped nor duplicated.
+  `verify()` after a run compares against the snapshot that was
+  migrated.
+- Re-running with a checkpoint that records a completed migration raises
+  `CheckpointError` instead of duplicating and dropping rows.
+- A batch committed just before a crash, but not yet checkpointed, is
+  not written twice on resume.
+- Pre-flight runs every check before it changes the destination, so
+  auto-added columns are no longer committed when a later check fails.
+- Writing to a destination partitioned by bucket, truncate, year, month,
+  day or hour without `pyiceberg-core` fails before any write with an
+  `ImportError` naming `pip install 'iceberg-ops[transforms]'`.
+- The first upsert after a schema-only change (for example an
+  auto-added column) no longer fails; an empty snapshot is committed
+  first, as a workaround for apache/iceberg-python#2467.
+- Composite-key upserts with more than a few thousand keys per batch no
+  longer crash the interpreter (apache/iceberg-python#3508). Large
+  batches are split into sub-batches of 1,000 keys within one commit.
+- When the source schema changed after the pinned snapshot, the plan
+  uses the schema at that snapshot.
+- `verify(method="partition_hash")` works when a partition field's name
+  differs from its source column, and raises `NotImplementedError` for
+  non-identity partitions instead of reporting a mismatch (the CLI
+  prints `Verify error:` and exits 1).
+- A resume raises `CheckpointError` when the destination was rolled
+  back past a checkpointed batch, instead of skipping that batch's
+  rows. Every refusal that follows committed batches names the
+  snapshot to roll back to before `reset()`.
+- Upsert keys that the plan would fill with null are rejected before
+  any write.
+- `DataMigrator` and `migrate-data` refuse a source and destination
+  that are the same table.
+- Checkpoint writes are fsync-ed before the atomic rename.
+
+Maintenance:
+
+- `remove_orphan_files` no longer deletes live position-delete,
+  equality-delete and deletion-vector files, or registered table and
+  partition statistics files.
+- `compact_small_files_streaming` no longer deletes its staged files
+  when the commit outcome is unknown (for example
+  `CommitStateUnknownException`), which could make a committed table
+  unreadable.
+- `expire_snapshots` skips branch and tag heads (including the current
+  snapshot after a rollback) instead of aborting, and dry runs select
+  the same snapshots as real runs. A max age older than the epoch
+  expires nothing instead of overflowing.
+- Compaction and orphan removal accept Windows drive-letter table
+  locations.
+
+### Security
+
+- Generated migration files cannot run injected code: the `-m`
+  message and column, partition and property names and values are
+  written as Python literals, and docstring text is escaped.
+- Polaris OAuth2 token-endpoint errors no longer echo the response
+  payload, which could leak tokens to logs or stderr. The client strips
+  surrounding whitespace from a token, rejects an empty token or one
+  with inner whitespace or control characters (`run-policies` exits
+  2), and keeps the token and client secret out of connection-error
+  messages.
+- A failed migration's history row holds only `"<Type>: <message>"`;
+  the traceback is logged locally at `DEBUG` level, since everyone with
+  catalog access can read the history table.
+- `remove_orphan_files` matches file names after Unicode NFC
+  normalisation and case folding, does not follow directory symlinks
+  or Windows junctions, and never deletes a file whose real path is
+  outside the table root.
+- Compaction escapes identity-partition values the way PyIceberg does
+  and refuses to stage files outside `<table>/data`, so a partition
+  value such as `../../x` cannot write elsewhere.
+- `run-policies` reads `ICEBERG_OPS_POLARIS_TOKEN`,
+  `ICEBERG_OPS_POLARIS_CLIENT_ID` and `ICEBERG_OPS_POLARIS_CLIENT_SECRET`
+  (preferred over flags, which show up in process listings) and warns
+  when `--polaris-url` uses plain http to a non-loopback host.
+
+### Known limitations
+
+- Concurrent runs for one namespace are not guarded, only detected: a
+  second run can execute `upgrade()` before it fails to record history.
+  Run one migration job per namespace at a time and keep `upgrade()`
+  idempotent.
+- The history table keeps the latest status of each version per
+  namespace, not one row per attempt. Earlier states are readable by
+  time travel on the history table until its snapshots expire.
+- `status`, `current`, `plan` and `history` print only and exit 0 even
+  when a migration is recorded as failed; there is no `check` command
+  yet. The runner methods print their reports and return `None`.
+- CLI autogenerate diffs columns only. Autogenerate does not emit
+  nullability or doc changes, blocks nested type changes from a
+  `pyiceberg.schema.Schema` target, and does not compare existing
+  nested columns of a `pyarrow.Schema` target (they are treated as
+  unchanged; the CLI names them in a warning). Each generated step commits on its own, so a failing step
+  leaves earlier steps applied.
+- Only SQLite and in-memory catalogs are integration-tested. Glue, REST,
+  Hive and S3 Tables catalogs are reached through PyIceberg but have not
+  been tested against live services.
+- Data migration: `overwrite_partition` is limited to one commit and
+  identity-partitioned destinations; the destination table must already
+  exist. `batch_rows` bounds the rows per commit, not peak memory, which
+  grows with the number and size of source data files
+  (apache/iceberg-python#2407); an upsert also loads the matched
+  destination rows. Duplicate keys within one batch raise `ValueError`,
+  and across batches the later row wins. `verify="sample"` can report a
+  false mismatch on larger tables, and `partition_hash` needs an
+  identity-partitioned or unpartitioned destination and reads both
+  tables in full.
+- Maintenance: compaction and orphan removal support the local
+  filesystem only. Compaction supports identity partitions and
+  unpartitioned tables (other partitions are skipped), rewrites
+  qualifying partitions whole, writes `2N + 1` snapshots for `N`
+  partitions, uses PyArrow's default codec, fails when a concurrent
+  append lands in a partition being compacted, and its memory is not bounded by `target_size_mb`
+  (apache/iceberg-python#2407). Delete files and deletion vectors are
+  out of scope for compaction (there is no guard) and the health
+  report. Snapshot expiry is metadata-only.
+- The Polaris executor does not call the Polaris 1.x policy endpoints,
+  so fetching policies from a released Polaris server fails. It has not
+  been validated against a live server.
+- Python 3.15 is not tested. At release time PyIceberg 0.12.0 published
+  no Python 3.15 wheels.
+- PyArrow 26 and later refuse to import next to NumPy 1.x. iceberg-ops
+  does not use NumPy; in an environment pinned to NumPy 1.x, also pin
+  `pyarrow<26`.
+- On Windows, write a local warehouse in `.pyiceberg.yaml` as
+  `file://C:/path`; PyArrow rejects the `file:///C:/path` form.
+
+[Unreleased]: https://github.com/namitkewat/iceberg-ops/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/namitkewat/iceberg-ops/releases/tag/v0.1.0

iceberg_ops-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Namit Kewat
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.