sql2iceberg 0.1.0__tar.gz

sql2iceberg-0.1.0/.gitignore

@@ -0,0 +1,31 @@
+__pycache__/
+*.egg-info/
+dist/
+build/
+.venv/
+.pytest_cache/
+out/
+.env
+jars/
+warehouse/
+seed/
+
+# personal / generated: kept local
+CLAUDE.md
+PUBLISHING-RUNBOOK.md
+medium-article-draft.md
+curated/
+converted/
+spark_scripts/
+scripts/cleanup_live_test.md
+catalog.db
+*.jsonl
+.venv/
+.ruff_cache/
+.env.*
+!.env.example
+scripts/curated_e2e.py
+scripts/export_spark.py
+scripts/pyspark_vs_sql.py
+scripts/pyspark_to_s3.py
+scripts/cost_compare.py

sql2iceberg-0.1.0/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 (first release, alpha)
+
+- Renamed from the working name `sf2iceberg` to `sql2iceberg`.
+- Detect SQL platform and SQL in `.sql`, `.py`, `.ipynb`; dialect-aware conversion to Spark SQL with safety findings.
+- Connectors: Snowflake (tested live); Postgres, MySQL, SQL Server, Oracle (experimental, mock-tested).
+- PySpark job generator: JDBC ingest to Iceberg, curation SQL, hadoop/jdbc/glue/rest/hive catalogs, S3 and GCS warehouses.
+- Publishers: Snowflake external Iceberg tables, AWS Glue/Athena, BigQuery BigLake. Plan first, `--apply` to change accounts.
+- `validate`: counts, non-null counts and sums across Snowflake, Athena and BigQuery.
+- `--trace` JSONL tracer with secret redaction; `doctor` and `issue` commands.
+- Live-tested on Snowflake, AWS and GCP with Iceberg format-version 2 (see docs/examples.md).

sql2iceberg-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,20 @@
+# Contributing
+
+## Setup
+```bash
+pip install -e ".[pytest,dev]"
+ruff check src tests
+pytest -q -m "not spark"          # no cloud accounts needed
+```
+Spark tests: `SQL2ICE_ICEBERG=1 pytest -m spark` with the Iceberg jars in `./jars` (see docs/environments.md).
+
+## Rules
+- Never commit credentials, account IDs, bucket names, or real company SQL or data. Examples use made-up data.
+- Untested platforms must stay labelled experimental in code and docs.
+- Anything that changes a cloud account must be gated behind an explicit `--apply` / `confirm=True`.
+
+## Reporting problems
+Open an issue with `sql2iceberg issue --open` (pre-fills your environment: names and versions only).
+
+## Security
+Do not open a public issue for a vulnerability. Use GitHub's "Report a vulnerability" (private advisory) on the repository.

sql2iceberg-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jeyaraman Ramalingam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sql2iceberg-0.1.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.5
+Name: sql2iceberg
+Version: 0.1.0
+Summary: Convert SQL from Snowflake and other databases to PySpark, persist it as Apache Iceberg tables, expose them to Snowflake, AWS and GCP, and validate the result.
+Project-URL: Homepage, https://github.com/jey1987-cmd/sql2iceberg
+Project-URL: Documentation, https://github.com/jey1987-cmd/sql2iceberg/tree/main/docs
+Project-URL: Issues, https://github.com/jey1987-cmd/sql2iceberg/issues
+Project-URL: Changelog, https://github.com/jey1987-cmd/sql2iceberg/blob/main/CHANGELOG.md
+Author: Jeyaraman Ramalingam
+License-Expression: MIT
+License-File: LICENSE
+Keywords: bigquery,data-engineering,glue,iceberg,migration,pyspark,snowflake,sql,sqlglot
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Code Generators
+Requires-Python: >=3.10
+Requires-Dist: sqlglot>=30
+Provides-Extra: all
+Requires-Dist: boto3>=1.34; extra == 'all'
+Requires-Dist: google-cloud-bigquery>=3.20; extra == 'all'
+Requires-Dist: pyspark>=3.5; extra == 'all'
+Requires-Dist: snowflake-connector-python>=3.12; extra == 'all'
+Provides-Extra: aws
+Requires-Dist: boto3>=1.34; extra == 'aws'
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: gcp
+Requires-Dist: google-cloud-bigquery>=3.20; extra == 'gcp'
+Provides-Extra: mssql
+Requires-Dist: pyodbc>=5; extra == 'mssql'
+Provides-Extra: mysql
+Requires-Dist: pymysql>=1.1; extra == 'mysql'
+Provides-Extra: oracle
+Requires-Dist: oracledb>=2; extra == 'oracle'
+Provides-Extra: postgres
+Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
+Provides-Extra: pytest
+Requires-Dist: chispa>=0.10; extra == 'pytest'
+Requires-Dist: pyspark>=3.5; extra == 'pytest'
+Requires-Dist: pytest>=8; extra == 'pytest'
+Provides-Extra: snowflake
+Requires-Dist: snowflake-connector-python>=3.12; extra == 'snowflake'
+Provides-Extra: spark
+Requires-Dist: pyspark>=3.5; extra == 'spark'
+Description-Content-Type: text/markdown
+
+# sql2iceberg
+
+Take the SQL in your code base, convert it to **PySpark**, persist the result as **Apache Iceberg** tables, expose those
+tables to **Snowflake, AWS (Glue/Athena) and GCP (BigQuery)**, and **validate** the migration.
+
+**Status: alpha (0.1.0).** One source platform (Snowflake) and three target platforms were exercised against real
+accounts; everything else is written but untested and labelled so. See [what was tested](docs/environments.md#what-was-tested)
+and [what is not supported yet](#limitations).
+
+```
+your code ──detect──▶ SQL ──convert──▶ Spark SQL ──generate──▶ PySpark job ──run──▶ Iceberg on S3 / GCS
+                                                                                      │
+                          validate ◀──── counts, nulls, sums ────────────────────────┤
+                                                                                      ├─▶ Snowflake (external Iceberg table)
+                                                                                      ├─▶ AWS Glue / Athena
+                                                                                      └─▶ BigQuery (BigLake)
+```
+
+## Core capabilities
+
+| # | Capability | Command / API | Status |
+|---|---|---|---|
+| 1 | **Detect** the SQL platform and find SQL in `.sql`, `.py`, `.ipynb` | `detect`, `detect_platform()` | Snowflake tested; other dialects parse via SQLGlot |
+| 2 | **Connect** to the source (you set up authentication first) | `connectors.get_connector()` | Snowflake tested live; Postgres, MySQL, SQL Server, Oracle **experimental** (mock-tested only) |
+| 3 | **Convert** to Spark SQL and flag what cannot be trusted | `convert`, `convert_statement()` | tested |
+| 4 | **Generate a PySpark job**: JDBC read, write Iceberg, run the curation SQL | `pyspark`, `pyspark_pipeline()` | tested end to end (Snowflake to Iceberg on S3 and on GCS) |
+| 5 | **Expose** Iceberg tables to other platforms | `publish`, `publish.*Publisher` | Snowflake, AWS Glue/Athena, BigQuery all tested live |
+| 6 | **Validate** counts, non-null counts and column sums across platforms | `validate`, `validate.validate()` | tested live across four platforms |
+| 7 | **Report**: converted PySpark, plan of changes, validation results (Markdown and JSON) | all commands | tested |
+| 8 | **Trace** every step with timings, secrets redacted | `--trace trace.jsonl`, `tracing.span()` | tested |
+| 9 | **Diagnose** your environment; open a pre-filled GitHub issue | `doctor`, `issue` | tested |
+
+Safety defaults: `publish` prints a plan and changes nothing unless you pass `--apply`; credentials are only read from
+environment variables or your cloud profile; generated scripts never contain secrets.
+
+## Install
+
+```bash
+pip install sql2iceberg                       # core: detect + convert (needs only sqlglot)
+pip install "sql2iceberg[spark]"              # run the generated PySpark jobs locally
+pip install "sql2iceberg[snowflake,aws,gcp]"  # connectors and publishers you need
+pip install "sql2iceberg[all]"                # spark + snowflake + aws + gcp
+pip install "sql2iceberg[pytest]"             # pytest plugin and fixtures
+```
+
+Extras: `spark`, `snowflake`, `postgres`, `mysql`, `mssql`, `oracle`, `aws`, `gcp`, `pytest`, `all`, `dev`.
+
+## Environment dependencies (summary)
+
+| Needed for | Requirement |
+|---|---|
+| everything | Python 3.10 or newer |
+| running generated Spark jobs | **Java 17** (Java 21 should work), `pyspark` 3.5 or 4.x. Do not set `SPARK_HOME` to another Spark. |
+| Iceberg tables | Iceberg Spark runtime jar matching your Spark (we used `iceberg-spark-runtime-4.1_2.13-1.12.0`) |
+| Iceberg on S3 / GCS | `iceberg-aws-bundle` / `iceberg-gcp-bundle` jars |
+| reading the source over JDBC | the JDBC driver jar (we used `snowflake-jdbc-3.19.0`) |
+| Snowflake | account, role that can create external volumes, catalog integrations and Iceberg tables; a warehouse |
+| AWS | an AWS profile or environment credentials with S3, Glue and (to count) Athena permissions |
+| GCP | `gcloud auth login`, a project **with billing enabled**, a BigQuery connection to the bucket |
+| behind a TLS-inspecting proxy | a CA bundle for Python and a Java truststore: see [troubleshooting](docs/troubleshooting.md) |
+
+Full details, versions we tested and the list of every package and jar: [docs/environments.md](docs/environments.md) and
+[docs/packages.md](docs/packages.md). Run `sql2iceberg doctor` to check your machine.
+
+## Quick start
+
+```bash
+# 1. which platform is this code base for, and what SQL is in it?
+sql2iceberg detect path/to/code
+# 2. convert, and see what needs hand work
+sql2iceberg convert path/to/code --out out
+# 3. generate the PySpark job (reads the source over JDBC, writes Iceberg, runs the curation SQL)
+sql2iceberg pyspark path/to/code --out out --platform snowflake \
+    --catalog-type jdbc --catalog-uri jdbc:sqlite:catalog.db \
+    --warehouse s3://my-bucket/warehouse --namespace live --iceberg-version 2
+# 4. run it (set SNOWFLAKE_USER and SNOWFLAKE_PRIVATE_KEY_PATH or SNOWFLAKE_PASSWORD first)
+spark-submit out/pipeline_pyspark.py
+# 5. expose the tables: read the plan first, then --apply
+sql2iceberg publish publish.json
+sql2iceberg publish publish.json --apply
+# 6. validate across platforms
+sql2iceberg validate validate.json --out out
+```
+
+Add `--trace trace.jsonl` to any command to record a timed trace of the run.
+Worked examples from real runs (configs, plans, results): [docs/examples.md](docs/examples.md).
+
+## Python API
+
+```python
+from sql2iceberg import detect_path, convert_statement
+from sql2iceberg.pyspark_gen import MigrationConfig, pyspark_pipeline
+from sql2iceberg.publish import SnowflakePublisher, SnowflakeTarget, tables_from_jdbc_catalog
+from sql2iceberg.validate import validate, Target, ColumnSpec
+
+stmts = detect_path("pipelines/", dialect="snowflake")
+script, report = pyspark_pipeline(stmts, MigrationConfig(platform="snowflake", warehouse="s3://b/wh",
+                                                         catalog_type="jdbc", catalog_uri="jdbc:sqlite:catalog.db"))
+```
+
+## Documentation
+
+- [Environments and what was tested](docs/environments.md)
+- [Every package, jar and service used in this build](docs/packages.md)
+- [Examples from the live tests](docs/examples.md)
+- [Troubleshooting frequent problems](docs/troubleshooting.md)
+- [Known conversion differences between Snowflake and Spark](docs/known-issues.md)
+- [Roadmap](docs/roadmap.md)
+
+## Getting help and reporting problems
+
+- Run `sql2iceberg doctor`, then `sql2iceberg issue --title "short summary" --open` to open a GitHub issue pre-filled with
+  your environment (names and versions only, never values or credentials).
+- Attach a trace (`--trace trace.jsonl`) after reading it. Secrets are redacted but check anyway.
+- Issues: <https://github.com/jey1987-cmd/sql2iceberg/issues>. Please read [troubleshooting](docs/troubleshooting.md) first.
+- Security problems: do not open a public issue; see [CONTRIBUTING.md](CONTRIBUTING.md#security).
+
+## Limitations
+
+- Only **Snowflake as a source** and **Snowflake, AWS Glue/Athena, BigQuery as targets** were run against real systems.
+  Postgres, MySQL, SQL Server and Oracle connectors, REST/Hive catalogs, Spark 3.5, Azure and Databricks are untested.
+- Tested with Iceberg **format-version 2**. Whether Snowflake and BigQuery read v3 tables is unverified here.
+- BigQuery cannot read an S3 warehouse. The GCP path writes a second Iceberg copy to GCS.
+- Snowflake object-store tables and BigQuery tables point at one metadata file (a snapshot): refresh after new writes.
+- Validation compares counts, non-null counts and sums. It does not compare every row.
+- Tested on Windows 10 with Python 3.13. Linux and macOS should work; CI covers Linux but live tests were not run there.
+- Conversion uses [SQLGlot](https://github.com/tobymao/sqlglot); the converter flags constructs that behave differently in
+  Spark (see [known issues](docs/known-issues.md)) but a clean result still needs your own verification.
+
+## License
+
+MIT. Not affiliated with Snowflake, the Apache Software Foundation, Amazon or Google. "Iceberg" is a trademark of the ASF.

sql2iceberg-0.1.0/README.md

@@ -0,0 +1,132 @@
+# sql2iceberg
+
+Take the SQL in your code base, convert it to **PySpark**, persist the result as **Apache Iceberg** tables, expose those
+tables to **Snowflake, AWS (Glue/Athena) and GCP (BigQuery)**, and **validate** the migration.
+
+**Status: alpha (0.1.0).** One source platform (Snowflake) and three target platforms were exercised against real
+accounts; everything else is written but untested and labelled so. See [what was tested](docs/environments.md#what-was-tested)
+and [what is not supported yet](#limitations).
+
+```
+your code ──detect──▶ SQL ──convert──▶ Spark SQL ──generate──▶ PySpark job ──run──▶ Iceberg on S3 / GCS
+                                                                                      │
+                          validate ◀──── counts, nulls, sums ────────────────────────┤
+                                                                                      ├─▶ Snowflake (external Iceberg table)
+                                                                                      ├─▶ AWS Glue / Athena
+                                                                                      └─▶ BigQuery (BigLake)
+```
+
+## Core capabilities
+
+| # | Capability | Command / API | Status |
+|---|---|---|---|
+| 1 | **Detect** the SQL platform and find SQL in `.sql`, `.py`, `.ipynb` | `detect`, `detect_platform()` | Snowflake tested; other dialects parse via SQLGlot |
+| 2 | **Connect** to the source (you set up authentication first) | `connectors.get_connector()` | Snowflake tested live; Postgres, MySQL, SQL Server, Oracle **experimental** (mock-tested only) |
+| 3 | **Convert** to Spark SQL and flag what cannot be trusted | `convert`, `convert_statement()` | tested |
+| 4 | **Generate a PySpark job**: JDBC read, write Iceberg, run the curation SQL | `pyspark`, `pyspark_pipeline()` | tested end to end (Snowflake to Iceberg on S3 and on GCS) |
+| 5 | **Expose** Iceberg tables to other platforms | `publish`, `publish.*Publisher` | Snowflake, AWS Glue/Athena, BigQuery all tested live |
+| 6 | **Validate** counts, non-null counts and column sums across platforms | `validate`, `validate.validate()` | tested live across four platforms |
+| 7 | **Report**: converted PySpark, plan of changes, validation results (Markdown and JSON) | all commands | tested |
+| 8 | **Trace** every step with timings, secrets redacted | `--trace trace.jsonl`, `tracing.span()` | tested |
+| 9 | **Diagnose** your environment; open a pre-filled GitHub issue | `doctor`, `issue` | tested |
+
+Safety defaults: `publish` prints a plan and changes nothing unless you pass `--apply`; credentials are only read from
+environment variables or your cloud profile; generated scripts never contain secrets.
+
+## Install
+
+```bash
+pip install sql2iceberg                       # core: detect + convert (needs only sqlglot)
+pip install "sql2iceberg[spark]"              # run the generated PySpark jobs locally
+pip install "sql2iceberg[snowflake,aws,gcp]"  # connectors and publishers you need
+pip install "sql2iceberg[all]"                # spark + snowflake + aws + gcp
+pip install "sql2iceberg[pytest]"             # pytest plugin and fixtures
+```
+
+Extras: `spark`, `snowflake`, `postgres`, `mysql`, `mssql`, `oracle`, `aws`, `gcp`, `pytest`, `all`, `dev`.
+
+## Environment dependencies (summary)
+
+| Needed for | Requirement |
+|---|---|
+| everything | Python 3.10 or newer |
+| running generated Spark jobs | **Java 17** (Java 21 should work), `pyspark` 3.5 or 4.x. Do not set `SPARK_HOME` to another Spark. |
+| Iceberg tables | Iceberg Spark runtime jar matching your Spark (we used `iceberg-spark-runtime-4.1_2.13-1.12.0`) |
+| Iceberg on S3 / GCS | `iceberg-aws-bundle` / `iceberg-gcp-bundle` jars |
+| reading the source over JDBC | the JDBC driver jar (we used `snowflake-jdbc-3.19.0`) |
+| Snowflake | account, role that can create external volumes, catalog integrations and Iceberg tables; a warehouse |
+| AWS | an AWS profile or environment credentials with S3, Glue and (to count) Athena permissions |
+| GCP | `gcloud auth login`, a project **with billing enabled**, a BigQuery connection to the bucket |
+| behind a TLS-inspecting proxy | a CA bundle for Python and a Java truststore: see [troubleshooting](docs/troubleshooting.md) |
+
+Full details, versions we tested and the list of every package and jar: [docs/environments.md](docs/environments.md) and
+[docs/packages.md](docs/packages.md). Run `sql2iceberg doctor` to check your machine.
+
+## Quick start
+
+```bash
+# 1. which platform is this code base for, and what SQL is in it?
+sql2iceberg detect path/to/code
+# 2. convert, and see what needs hand work
+sql2iceberg convert path/to/code --out out
+# 3. generate the PySpark job (reads the source over JDBC, writes Iceberg, runs the curation SQL)
+sql2iceberg pyspark path/to/code --out out --platform snowflake \
+    --catalog-type jdbc --catalog-uri jdbc:sqlite:catalog.db \
+    --warehouse s3://my-bucket/warehouse --namespace live --iceberg-version 2
+# 4. run it (set SNOWFLAKE_USER and SNOWFLAKE_PRIVATE_KEY_PATH or SNOWFLAKE_PASSWORD first)
+spark-submit out/pipeline_pyspark.py
+# 5. expose the tables: read the plan first, then --apply
+sql2iceberg publish publish.json
+sql2iceberg publish publish.json --apply
+# 6. validate across platforms
+sql2iceberg validate validate.json --out out
+```
+
+Add `--trace trace.jsonl` to any command to record a timed trace of the run.
+Worked examples from real runs (configs, plans, results): [docs/examples.md](docs/examples.md).
+
+## Python API
+
+```python
+from sql2iceberg import detect_path, convert_statement
+from sql2iceberg.pyspark_gen import MigrationConfig, pyspark_pipeline
+from sql2iceberg.publish import SnowflakePublisher, SnowflakeTarget, tables_from_jdbc_catalog
+from sql2iceberg.validate import validate, Target, ColumnSpec
+
+stmts = detect_path("pipelines/", dialect="snowflake")
+script, report = pyspark_pipeline(stmts, MigrationConfig(platform="snowflake", warehouse="s3://b/wh",
+                                                         catalog_type="jdbc", catalog_uri="jdbc:sqlite:catalog.db"))
+```
+
+## Documentation
+
+- [Environments and what was tested](docs/environments.md)
+- [Every package, jar and service used in this build](docs/packages.md)
+- [Examples from the live tests](docs/examples.md)
+- [Troubleshooting frequent problems](docs/troubleshooting.md)
+- [Known conversion differences between Snowflake and Spark](docs/known-issues.md)
+- [Roadmap](docs/roadmap.md)
+
+## Getting help and reporting problems
+
+- Run `sql2iceberg doctor`, then `sql2iceberg issue --title "short summary" --open` to open a GitHub issue pre-filled with
+  your environment (names and versions only, never values or credentials).
+- Attach a trace (`--trace trace.jsonl`) after reading it. Secrets are redacted but check anyway.
+- Issues: <https://github.com/jey1987-cmd/sql2iceberg/issues>. Please read [troubleshooting](docs/troubleshooting.md) first.
+- Security problems: do not open a public issue; see [CONTRIBUTING.md](CONTRIBUTING.md#security).
+
+## Limitations
+
+- Only **Snowflake as a source** and **Snowflake, AWS Glue/Athena, BigQuery as targets** were run against real systems.
+  Postgres, MySQL, SQL Server and Oracle connectors, REST/Hive catalogs, Spark 3.5, Azure and Databricks are untested.
+- Tested with Iceberg **format-version 2**. Whether Snowflake and BigQuery read v3 tables is unverified here.
+- BigQuery cannot read an S3 warehouse. The GCP path writes a second Iceberg copy to GCS.
+- Snowflake object-store tables and BigQuery tables point at one metadata file (a snapshot): refresh after new writes.
+- Validation compares counts, non-null counts and sums. It does not compare every row.
+- Tested on Windows 10 with Python 3.13. Linux and macOS should work; CI covers Linux but live tests were not run there.
+- Conversion uses [SQLGlot](https://github.com/tobymao/sqlglot); the converter flags constructs that behave differently in
+  Spark (see [known issues](docs/known-issues.md)) but a clean result still needs your own verification.
+
+## License
+
+MIT. Not affiliated with Snowflake, the Apache Software Foundation, Amazon or Google. "Iceberg" is a trademark of the ASF.

sql2iceberg-0.1.0/docs/environments.md

@@ -0,0 +1,65 @@
+# Environments
+
+## Requirements
+
+| Component | Required | Notes |
+|---|---|---|
+| Python | 3.10 or newer | declared in `pyproject.toml`; live-tested on 3.13 |
+| Java | 17 (21 should work) for PySpark 4; 8, 11 or 17 for PySpark 3.5 | `JAVA_HOME` must point at it. `sql2iceberg doctor` warns if `JAVA_HOME` and `java` on `PATH` differ |
+| PySpark | 3.5 or 4.x | live-tested on 4.1.4. Do not set `SPARK_HOME` to a different Spark |
+| Iceberg Spark runtime jar | must match your Spark and Scala version | tested: `iceberg-spark-runtime-4.1_2.13-1.12.0.jar` |
+| Iceberg cloud bundles | `iceberg-aws-bundle` for S3 and Glue; `iceberg-gcp-bundle` for GCS | tested: 1.12.0 both |
+| JDBC driver for the source | one jar per source platform | tested: `snowflake-jdbc-3.19.0.jar`; `sqlite-jdbc-3.46.1.3.jar` for the local catalog and unit tests |
+| Network | Maven Central (or local jars), your cloud endpoints | behind a proxy that re-signs TLS see [troubleshooting](troubleshooting.md#certificate_unknown--ssl-errors) |
+
+The jars can come from Maven (`--packages`, `spark.jars.packages`) or from local files (`MigrationConfig.jars`).
+On locked-down networks download them once and use local files.
+
+## Authentication prerequisites (you do these before running)
+
+| Platform | What to set up | Variables / tools read |
+|---|---|---|
+| Snowflake | a user with key-pair (recommended), password or SSO; a role and warehouse | `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, `SNOWFLAKE_PRIVATE_KEY_PATH` or `SNOWFLAKE_PASSWORD` or `SNOWFLAKE_AUTHENTICATOR`, `SNOWFLAKE_ROLE`, `SNOWFLAKE_WAREHOUSE` |
+| Snowflake to S3 | an external volume with a storage integration and IAM role that Snowflake can assume | created once in your account; the library can create it if you pass `aws_role_arn` |
+| AWS | profile or environment credentials | `AWS_PROFILE`, `AWS_REGION`, optionally `AWS_CA_BUNDLE` |
+| GCP | `gcloud auth login`; a project with **billing enabled**; BigQuery Connection API enabled | `gcloud` credentials; a short-lived token is passed to Spark through `GCS_OAUTH_TOKEN` |
+| Postgres, MySQL, SQL Server, Oracle (experimental) | native client environment variables | `PGHOST` ..., `MYSQL_HOST` ..., `MSSQL_SERVER` ..., `ORACLE_DSN` ... |
+
+## Supported environments
+
+| Area | Verified live | Written, mock-tested only | Not supported yet |
+|---|---|---|---|
+| Source SQL platform | Snowflake | Postgres, MySQL, SQL Server (T-SQL), Oracle | others |
+| Spark | PySpark 4.1.4 (local mode) | PySpark 3.5 | Databricks, EMR, Dataproc submit helpers |
+| Iceberg catalog | JDBC catalog (SQLite file), Hadoop (local), Glue (for `register_table`) | REST, Hive | Nessie, Polaris |
+| Object storage | S3 (us-east-2), GCS (US) | | Azure Blob / ADLS |
+| Target: Snowflake | external Iceberg tables via object-store catalog integration, existing external volume | Glue-integration mode, creating volumes and integrations | Snowflake-managed Iceberg writes from Spark |
+| Target: AWS | Glue Data Catalog registration and Athena queries | | Redshift Spectrum, Lake Formation |
+| Target: GCP | BigQuery BigLake Iceberg tables over GCS | | BigQuery Omni on S3 |
+| Iceberg format version | 2 | 3 (generated by default, unverified on Snowflake and BigQuery) | |
+| Operating system | Windows 10 | CI on Ubuntu (unit tests) | macOS |
+
+## What was tested
+
+The live run on 2026-11-09 used a synthetic data set ("corpus": customers, orders, order items) and these exact versions.
+
+| Item | Version / value |
+|---|---|
+| OS | Windows 10 (10.0.19045), Git Bash and PowerShell |
+| Python | 3.13.17 |
+| Java | Eclipse Temurin 17.0.21 |
+| PySpark | 4.1.4 (Spark local mode) |
+| Iceberg | 1.12.0 (`spark-runtime-4.1_2.13`, `aws-bundle`, `gcp-bundle`) |
+| SQLGlot | 30.30.0 |
+| snowflake-connector-python | 5.2.0 |
+| Snowflake JDBC | 3.19.0 |
+| Snowflake | one account on AWS us-east-2, role SYSADMIN, key-pair login, XSMALL warehouse |
+| AWS | S3, Glue Data Catalog, Athena workgroup `primary`, region us-east-2, boto3 1.43.127, AWS CLI 2.30.6 |
+| GCP | one project, BigQuery location `US`, GCS location `US`, Cloud SDK 556.0.0 |
+
+Results (all matched the Snowflake source): 20 customers, 100 orders, 100 order items, 75 curated orders on
+Snowflake (Iceberg on S3), AWS Athena and BigQuery (Iceberg on GCS). Step by step with commands:
+[examples.md](examples.md).
+
+The unit test suite (`pytest -m "not spark"`, 75 tests) needs no cloud accounts. One test runs the generated PySpark
+job against SQLite with a local Iceberg catalog (`SQL2ICE_ICEBERG=1` and the jars in `./jars`).