crowdb-tpc-loader 0.1.0__tar.gz

crowdb_tpc_loader-0.1.0/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+- Load TPC-H and TPC-DS datasets into Iceberg tables, with validation and recoverable reports.
+- Write separate tables concurrently, with 8 workers by default and a configurable 1–24 range.
+- Retry transient Catalog 503 responses during table creation without retrying ambiguous table commits.
+- Verify the published CROWDB Iceberg image with a small TPC-H load and DuckDB Q1 read.
+- Add CI checks and a manually triggered PyPI release workflow.

crowdb_tpc_loader-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

crowdb_tpc_loader-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE README.md CHANGELOG.md
+recursive-include docs *.md
+recursive-include scripts *.py
+recursive-include tests *.py
+include requirements-integration.txt

crowdb_tpc_loader-0.1.0/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: crowdb-tpc-loader
+Version: 0.1.0
+Summary: Generate and safely load TPC-H and TPC-DS Parquet datasets into an Iceberg REST catalog
+Author: CrowDB contributors
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/buzzcrow/crowdb-tpc-loader
+Project-URL: Issues, https://github.com/buzzcrow/crowdb-tpc-loader/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Requires-Python: <3.13,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: botocore<2,>=1.34
+Requires-Dist: pyarrow<24,>=18
+Requires-Dist: duckdb<1.6,>=1.4
+Requires-Dist: pyiceberg[pyarrow]<0.11,>=0.10
+Requires-Dist: packaging<27,>=24
+Provides-Extra: dev
+Requires-Dist: pytest<10,>=8; extra == "dev"
+Requires-Dist: pytest-cov<8,>=5; extra == "dev"
+Requires-Dist: build<2,>=1.2; extra == "dev"
+Requires-Dist: ruff<1,>=0.11; extra == "dev"
+Requires-Dist: twine<7,>=6; extra == "dev"
+Provides-Extra: sql-test
+Requires-Dist: sqlalchemy<3,>=2; extra == "sql-test"
+Dynamic: license-file
+
+# CrowDB TPC Loader
+
+Generate TPC-H or TPC-DS Parquet, upload it to CROWDB Iceberg, and register complete tables through the REST Catalog. The loader creates 8 TPC-H or 24 TPC-DS tables. It does not run benchmark SQL queries or claim TPC certification.
+
+- Repository: [buzzcrow/crowdb-tpc-loader](https://github.com/buzzcrow/crowdb-tpc-loader)
+- End-to-end guide: [CROWDB TPC loader documentation](https://crowdb.dev/docs/tpc-loader/)
+
+## Install
+
+Python 3.10–3.12 is required. After the first PyPI release:
+
+```sh
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install crowdb-tpc-loader
+```
+
+Until PyPI publication, install from a checkout with `python -m pip install --only-binary=:all: -e .`. The first TPC-H run may download `tpchgen-cli` 3.0.0; TPC-DS may download DuckDB's `tpcds` extension. Use `--no-download` and provide these components ahead of time for an offline run.
+
+## Load
+
+Start `crowdb/crowdb-iceberg:latest` and export the `ICEBERG_URI` and `ICEBERG_TOKEN` values printed by `docker exec <container> crowdb-monitor credentials show --format env`. Keep the token private. Use a new namespace for each run:
+
+```sh
+crowdb-tpc-loader load --benchmark tpch --sf 0.01 \
+  --namespace tpch_demo --report-file ./tpch-demo.json
+crowdb-tpc-loader load --benchmark tpcds --sf 0.01 \
+  --namespace tpcds_demo --report-file ./tpcds-demo.json
+```
+
+The loader validates the entire generated dataset before creating a table. It writes different tables concurrently, with 8 workers by default. Use `--upload-workers N` to control concurrent Iceberg table writes (1–24). Each table's files are uploaded and registered in one snapshot, with a durable report checkpoint before each remote side effect. One table's failure does not roll back tables that already succeeded. The report identifies committed, unregistered, and uncertain files; see [recovery](docs/RECOVERY.md) before retrying. An existing table stops the default load; `--on-exists skip` leaves it unchanged without verifying it.
+
+For local Parquet only, use `crowdb-tpc-loader generate --benchmark tpch --sf 0.01 --output-dir ./tpch-001`.
+
+## Check the result
+
+Run the read-only verifier from a checkout after loading:
+
+```sh
+python scripts/verify_crowdb.py ./tpch-demo.json --require-complete --iceberg-scan
+```
+
+With DuckDB's `iceberg` and `httpfs` extensions, attach the REST Catalog using its token and query `tpch_demo.region` or run TPC-H Q1 against `tpch_demo.lineitem`. The [website guide](https://crowdb.dev/docs/tpc-loader/) has the SQL. A published `latest` image and the local DuckDB 1.5.6 CLI passed an SF 0.01 TPC-H import, independent table verification, and Q1 read on October 1, 2026. This is an integration check, not a performance result. See [compatibility](docs/COMPATIBILITY.md) and the [test record](docs/TEST_REPORT.md).
+
+## Develop and publish
+
+```sh
+python -m pip install --only-binary=:all: -e '.[dev,sql-test]'
+ruff check src tests scripts
+ruff format --check src tests scripts
+python -m pytest
+python -m build
+python -m twine check dist/*
+```
+
+CI runs lint, format, tests, and package checks. Publishing is manual through [the PyPI workflow](.github/workflows/publish.yml) from a `release/v<version>` branch after configuring PyPI Trusted Publishing. See [testing](docs/TESTING.md) for optional real generator and CROWDB runs. The package is Apache-2.0; third-party generators and libraries keep their own licenses.
+
+To publish `0.1.0`:
+
+1. In PyPI, create a pending Trusted Publisher for project `crowdb-tpc-loader`: GitHub owner `buzzcrow`, repository `crowdb-tpc-loader`, workflow `publish.yml`, environment `pypi`. Create the `pypi` environment in GitHub.
+2. After CI is green, create and push branch `release/v0.1.0` from the commit to publish.
+3. In GitHub Actions, open **Publish to PyPI**, click **Run workflow**, select branch `release/v0.1.0`, then run it. The workflow verifies the branch name against the package version, reruns checks, builds distributions, and publishes through OIDC. No PyPI API token is stored in GitHub.
+4. Confirm the files on [PyPI](https://pypi.org/project/crowdb-tpc-loader/), then test `python -m pip install --no-cache-dir crowdb-tpc-loader==0.1.0` in a clean environment and run `crowdb-tpc-loader --version`.

crowdb_tpc_loader-0.1.0/README.md

@@ -0,0 +1,63 @@
+# CrowDB TPC Loader
+
+Generate TPC-H or TPC-DS Parquet, upload it to CROWDB Iceberg, and register complete tables through the REST Catalog. The loader creates 8 TPC-H or 24 TPC-DS tables. It does not run benchmark SQL queries or claim TPC certification.
+
+- Repository: [buzzcrow/crowdb-tpc-loader](https://github.com/buzzcrow/crowdb-tpc-loader)
+- End-to-end guide: [CROWDB TPC loader documentation](https://crowdb.dev/docs/tpc-loader/)
+
+## Install
+
+Python 3.10–3.12 is required. After the first PyPI release:
+
+```sh
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install crowdb-tpc-loader
+```
+
+Until PyPI publication, install from a checkout with `python -m pip install --only-binary=:all: -e .`. The first TPC-H run may download `tpchgen-cli` 3.0.0; TPC-DS may download DuckDB's `tpcds` extension. Use `--no-download` and provide these components ahead of time for an offline run.
+
+## Load
+
+Start `crowdb/crowdb-iceberg:latest` and export the `ICEBERG_URI` and `ICEBERG_TOKEN` values printed by `docker exec <container> crowdb-monitor credentials show --format env`. Keep the token private. Use a new namespace for each run:
+
+```sh
+crowdb-tpc-loader load --benchmark tpch --sf 0.01 \
+  --namespace tpch_demo --report-file ./tpch-demo.json
+crowdb-tpc-loader load --benchmark tpcds --sf 0.01 \
+  --namespace tpcds_demo --report-file ./tpcds-demo.json
+```
+
+The loader validates the entire generated dataset before creating a table. It writes different tables concurrently, with 8 workers by default. Use `--upload-workers N` to control concurrent Iceberg table writes (1–24). Each table's files are uploaded and registered in one snapshot, with a durable report checkpoint before each remote side effect. One table's failure does not roll back tables that already succeeded. The report identifies committed, unregistered, and uncertain files; see [recovery](docs/RECOVERY.md) before retrying. An existing table stops the default load; `--on-exists skip` leaves it unchanged without verifying it.
+
+For local Parquet only, use `crowdb-tpc-loader generate --benchmark tpch --sf 0.01 --output-dir ./tpch-001`.
+
+## Check the result
+
+Run the read-only verifier from a checkout after loading:
+
+```sh
+python scripts/verify_crowdb.py ./tpch-demo.json --require-complete --iceberg-scan
+```
+
+With DuckDB's `iceberg` and `httpfs` extensions, attach the REST Catalog using its token and query `tpch_demo.region` or run TPC-H Q1 against `tpch_demo.lineitem`. The [website guide](https://crowdb.dev/docs/tpc-loader/) has the SQL. A published `latest` image and the local DuckDB 1.5.6 CLI passed an SF 0.01 TPC-H import, independent table verification, and Q1 read on October 1, 2026. This is an integration check, not a performance result. See [compatibility](docs/COMPATIBILITY.md) and the [test record](docs/TEST_REPORT.md).
+
+## Develop and publish
+
+```sh
+python -m pip install --only-binary=:all: -e '.[dev,sql-test]'
+ruff check src tests scripts
+ruff format --check src tests scripts
+python -m pytest
+python -m build
+python -m twine check dist/*
+```
+
+CI runs lint, format, tests, and package checks. Publishing is manual through [the PyPI workflow](.github/workflows/publish.yml) from a `release/v<version>` branch after configuring PyPI Trusted Publishing. See [testing](docs/TESTING.md) for optional real generator and CROWDB runs. The package is Apache-2.0; third-party generators and libraries keep their own licenses.
+
+To publish `0.1.0`:
+
+1. In PyPI, create a pending Trusted Publisher for project `crowdb-tpc-loader`: GitHub owner `buzzcrow`, repository `crowdb-tpc-loader`, workflow `publish.yml`, environment `pypi`. Create the `pypi` environment in GitHub.
+2. After CI is green, create and push branch `release/v0.1.0` from the commit to publish.
+3. In GitHub Actions, open **Publish to PyPI**, click **Run workflow**, select branch `release/v0.1.0`, then run it. The workflow verifies the branch name against the package version, reruns checks, builds distributions, and publishes through OIDC. No PyPI API token is stored in GitHub.
+4. Confirm the files on [PyPI](https://pypi.org/project/crowdb-tpc-loader/), then test `python -m pip install --no-cache-dir crowdb-tpc-loader==0.1.0` in a clean environment and run `crowdb-tpc-loader --version`.

crowdb_tpc_loader-0.1.0/docs/COMPATIBILITY.md

@@ -0,0 +1,13 @@
+# Compatibility
+
+The package requires Python 3.10–3.12, PyArrow `>=18,<24`, PyIceberg `>=0.10,<0.11`, and DuckDB `>=1.4,<1.6`. TPC-H uses `tpchgen-cli` 3.x; automatic download pins 3.0.0. These ranges are package constraints, not a claim that every version combination passed. The [test record](TEST_REPORT.md) names the observed local combination.
+
+`load` requires an Iceberg REST Catalog and working remote FileIO. It reads table locations and credentials from the catalog; it never registers a local path. The default `crowdb_tpc_loader.crowdb_fileio.CrowdbFileIO` uses exact-object opens on CROWDB's native Iceberg endpoint, which does not require S3 prefix listing. Other deployments can provide a trusted FileIO class with `--py-io-impl`.
+
+Before generation, the loader checks catalog access, existing tables, namespace creation, and a staged FileIO write probe where cleanup is supported. CROWDB's native FileIO has no remote file delete, so the first real table upload acts as its write test. The entire generated dataset passes schema and footer checks before any benchmark table is created.
+
+The catalog can return transient HTTP 503 when its bounded operation capacity is busy. Table creation retries up to six attempts with short backoff. If a retry finds a table created by an earlier attempt in the same run, it verifies the run ID before continuing. Registration (`add_files`) is never blindly retried after an ambiguous response; the loader reads the current snapshot to reconcile the outcome. Keep the JSON report for [failure recovery](RECOVERY.md).
+
+Each worker owns a separate Catalog client and one table. Eight workers run by default; `--upload-workers` accepts 1–24. Files within a table upload in sequence, then that table commits once. Other tables can upload or commit at the same time. This can expose server backpressure; choose fewer workers for a small deployment if bounded retries are still exhausted. A failure in one table does not roll back another table's successful snapshot.
+
+TPC-DS generation uses DuckDB's `tpcds` extension and a disk database. `--memory-limit` controls DuckDB memory, not total process memory. TPC-H defaults to `ceil(SF / 10)` parts. Reports record the actual dependency and generator versions for each run. Distributed deployments and the full TPC query suites need separate acceptance.

crowdb_tpc_loader-0.1.0/docs/RECOVERY.md

@@ -0,0 +1,49 @@
+# Failure recovery: determine commit status before handling files
+
+The tool does not provide automatic resume, overwrite, drop, purge, or remote garbage collection. There is no atomic transaction across all 8 or 24 tables. With concurrent table writes, each table uploads its files and commits one snapshot independently. An upload or commit failure in one table does not roll back successful snapshots in other tables.
+
+## Reports and directories
+
+Keep the JSON report and working subdirectory shown in the console. A persistent `--report-file` path is the most reliable choice. Before uploading, the tool records the URI, run ID, and file status on disk. Before a commit, it marks the file `commit_unknown` and persists that state. If the process crashes between upload and commit, the unknown result will not be treated as definitely uncommitted.
+
+Forced process termination, sudden power loss, and disk damage can still interrupt the latest checkpoint. Atomic replacement, fsync, and reports in two locations reduce this risk but do not provide a distributed transaction guarantee.
+
+| Status | Meaning | Handling |
+|---|---|---|
+| `local` | No target remote URI yet | A locally generated file; this does not imply upload |
+| `upload_started` | Target URI recorded; the object may be absent, partly uploaded, or complete | Check whether the object exists; do not infer its size from status alone |
+| `uploaded_unregistered` | Upload accepted by the server; commit not attempted, or explicitly rejected and verified unregistered | Check other snapshots and references before cleanup |
+| `commit_unknown` | Commit started or may have started; result cannot be confirmed | Keep the file; do not blindly repeat `add_files` |
+| `registered` | URI observed in the verified current snapshot | Do not delete directly; follow the Iceberg snapshot and metadata lifecycle |
+
+`summary.unregistered_uploads` lists **candidates for inspection**, not objects that can be deleted in bulk. Keep `summary.uncertain_uploads` as a priority. `summary.tables_requiring_inspection` includes tables that may have been created but not fully verified, or for which writes failed. `table_creation_state=unknown_or_unvalidated` does not mean the table is definitely absent.
+
+## Common cases
+
+### Generation or schema validation fails
+
+This release creates benchmark tables only after validating the entire Parquet dataset. Check the generator version, actual schema, missing tables, and source-versus-export row counts in the report. Run again with a new output directory or namespace; do not overwrite the original files.
+
+### FileIO preflight fails
+
+A reachable REST endpoint does not prove that storage is writable. Check whether the returned storage address is reachable from the client, whether authentication needs a separate token, whether the correct FileIO was selected, and whether the HTTP service supports the required operations. Preflight uses an uncommitted staged table and does not fall back to a local path.
+
+### Some tables succeed and a later table fails
+
+First run the independent verifier on tables marked `succeeded` in the report. The failed table may exist but be empty, or it may have a committed or uncertain snapshot. A default rerun stops on existing tables as a protective measure.
+
+`--on-exists skip` does not repair a failed table and may skip an empty one. Do not use it to claim that the full dataset has been completed. The easiest way to verify a fresh run is to use a new namespace, then handle old data according to Iceberg's rules. If the old namespace must be kept, inspect every snapshot, branch, and reference for the table before deciding how to handle empty tables or orphaned objects. The tool does not perform these destructive operations.
+
+### Commit request times out
+
+The tool makes up to three rounds of read-only verification without calling `add_files` again. It records success only if the current snapshot's file set, row counts, and sizes match exactly. Otherwise, it records `uncertain` and marks that table uncertain; other in-flight tables may still finish.
+
+Another writer may subsequently change the final snapshot. Files recorded as `registered` may also be referenced by historical snapshots; absence from the current snapshot does not prove absence from history. Use the namespace, table UUID, run ID, and all file URIs to inspect server logs, snapshots, and branch references before proceeding.
+
+### Insufficient local space or process interruption
+
+Check the exit code, external report, and working path. The working directory is kept after a failure, and the tool does not delete remote benchmark objects. If local cleanup fails after a successful run, the tool records a cleanup warning without changing a verified successful remote commit into a failure. Before manually deleting a local directory, confirm that it is this run's working subdirectory, not the `--work-dir` parent.
+
+## Information to include in a bug report
+
+Keep the command with the token removed, Python/dependency/generator versions, run ID, error from the report, failed table, and FileIO type. Inspect the redacted report manually before sharing it. Do not include tokens, storage keys, signed URLs, complete environment variables, or private business data in an issue.

crowdb_tpc_loader-0.1.0/docs/TESTING.md

@@ -0,0 +1,37 @@
+# Testing
+
+CI runs `ruff check src tests scripts`, `ruff format --check src tests scripts`, and `python -m pytest` on Python 3.10, 3.11, and 3.12. It builds and checks the wheel and sdist on Python 3.12. The default tests include unit and local PyArrow/PyIceberg integration tests. Real generators and a CROWDB instance are opt-in.
+
+## Local checks
+
+```sh
+python -m pip install --only-binary=:all: -e '.[dev,sql-test]'
+ruff check src tests scripts
+ruff format --check src tests scripts
+python -m pytest -q
+python -m build
+python -m twine check dist/*
+```
+
+Check a built wheel in a fresh virtual environment before release. Confirm both `crowdb-tpc-loader --version` and `crowdb-tpc-loader load --help` work.
+
+## Real generators
+
+These may download generator binaries or extensions and need disk space:
+
+```sh
+CROWDB_TPC_GENERATOR_TESTS=1 python -m pytest tests/integration/test_generators.py -ra
+```
+
+Set `CROWDB_TPC_SF1_TESTS=1` as well to include full SF 1 datasets. Or run `python scripts/smoke_test.py --benchmark tpch --sf 0.01` for local generation only.
+
+## Real CROWDB load
+
+Use a dedicated disposable instance with `ICEBERG_URI` and `ICEBERG_TOKEN` exported. The tests create fresh remote namespaces and do not delete them:
+
+```sh
+CROWDB_TPC_CROWDB_TESTS=1 CROWDB_TPC_SF1_TESTS=1 \
+  python -m pytest tests/integration/test_crowdb.py -s -ra
+```
+
+For a smaller remote run, use `python scripts/smoke_test.py --benchmark tpch --sf 0.01 --load`. Preserve the JSON report on failure. A skipped integration test is not evidence of remote acceptance.

crowdb_tpc_loader-0.1.0/docs/TEST_REPORT.md

@@ -0,0 +1,10 @@
+# Test record
+
+## Published image, October 1, 2026
+
+- Image: `crowdb/crowdb-iceberg:latest`, digest `sha256:e4e7f80367094a35ab54c2bf1b0a9780811f9a668bc6037cfe7ced1b23373718`, Linux amd64 single-node.
+- Loader: eight concurrent table writes (`--upload-workers 8`), TPC-H SF 0.01, fresh namespace. All 8 tables committed; independent verifier checked 86,805 rows in manifests and remote Parquet footers, plus sample Iceberg scans after local staging was removed.
+- DuckDB CLI: locally built 1.5.6 with `iceberg` and `httpfs`. `region` returned `(1, AMERICA)`. Standard TPC-H Q1 on the imported `lineitem` returned four groups: `(A,F,14876)`, `(N,F,348)`, `(N,O,29181)`, `(R,F,14902)` for `(returnflag, linestatus, count_order)`. The CLI reported 0.075 seconds for that small warm-host run. No baseline or repeated trials were taken.
+- Initial 8-worker attempt saw four transient Catalog HTTP 503 responses and partial table success. The loader now retries bounded 503 create responses; a fresh namespace completed. Both reports are local test artifacts, not included in the package.
+
+This proves one integration path and one Q1 execution. It does not establish the full TPC-H suite, benchmark performance, distributed behavior, or TPC certification.

crowdb_tpc_loader-0.1.0/pyproject.toml

@@ -0,0 +1,64 @@
+[build-system]
+requires = ["setuptools>=77,<83"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "crowdb-tpc-loader"
+version = "0.1.0"
+description = "Generate and safely load TPC-H and TPC-DS Parquet datasets into an Iceberg REST catalog"
+readme = "README.md"
+requires-python = ">=3.10,<3.13"
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+authors = [{name = "CrowDB contributors"}]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Database",
+]
+dependencies = [
+  "botocore>=1.34,<2",
+  "pyarrow>=18,<24",
+  "duckdb>=1.4,<1.6",
+  "pyiceberg[pyarrow]>=0.10,<0.11",
+  "packaging>=24,<27",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8,<10", "pytest-cov>=5,<8", "build>=1.2,<2", "ruff>=0.11,<1", "twine>=6,<7"]
+sql-test = ["sqlalchemy>=2,<3"]
+
+[project.scripts]
+crowdb-tpc-loader = "crowdb_tpc_loader.cli:main"
+
+[project.urls]
+Repository = "https://github.com/buzzcrow/crowdb-tpc-loader"
+Issues = "https://github.com/buzzcrow/crowdb-tpc-loader/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+addopts = "-ra"
+markers = [
+  "integration: requires installed runtime dependencies",
+  "generator: invokes the real generator (may download binaries/extensions)",
+  "crowdb: writes a fresh namespace in an explicitly configured CrowDB instance",
+]
+
+[tool.ruff]
+line-length = 110
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/integration/test_iceberg_http.py" = ["E402", "F811"]
+"tests/integration/test_parquet.py" = ["E402"]

crowdb_tpc_loader-0.1.0/requirements-integration.txt

@@ -0,0 +1,11 @@
+# Candidate reproducibility matrix, NOT an already-tested dependency lock.
+# Install with --only-binary=:all: and run the real integration suite.
+pyarrow==22.0.0
+pyiceberg[pyarrow]==0.10.0
+duckdb==1.5.6
+packaging==25.0
+sqlalchemy==2.0.43
+pytest==8.4.2
+pytest-cov==6.2.1
+# Optional: use the loader's binary-only bootstrap instead of installing this.
+# tpchgen-cli==3.0.0

crowdb_tpc_loader-0.1.0/scripts/smoke_test.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""Run a local generation smoke test, or an explicitly requested remote load.
+
+No remote writes unless --load is present. Uses a unique output directory and,
+for loads, a unique namespace. It never drops remote data.
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+import subprocess
+import sys
+import uuid
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--benchmark", choices=("tpch", "tpcds"), required=True)
+    parser.add_argument("--sf", default="0.01")
+    parser.add_argument("--output-root", type=Path, default=Path("smoke-results"))
+    parser.add_argument(
+        "--load", action="store_true", help="explicitly enable writes to ICEBERG_URI using ICEBERG_TOKEN"
+    )
+    parser.add_argument("--py-io-impl")
+    parser.add_argument("--tpchgen", type=Path)
+    parser.add_argument("--full-read", action="store_true")
+    args = parser.parse_args()
+    run = args.output_root / (args.benchmark + "-" + uuid.uuid4().hex[:12])
+    run.mkdir(parents=True, exist_ok=False)
+    command = [
+        sys.executable,
+        "-m",
+        "crowdb_tpc_loader",
+        "load" if args.load else "generate",
+        "--benchmark",
+        args.benchmark,
+        "--sf",
+        args.sf,
+        "--report-file",
+        str(run / "report.json"),
+    ]
+    if args.tpchgen:
+        command += ["--tpchgen", str(args.tpchgen)]
+    if args.load:
+        namespace = "smoke_" + args.benchmark + "_" + uuid.uuid4().hex[:12]
+        command += ["--namespace", namespace, "--work-dir", str(run / "staging")]
+        if args.py_io_impl:
+            command += ["--py-io-impl", args.py_io_impl]
+    else:
+        command += ["--output-dir", str(run / "generated")]
+    print(f"Smoke output: {run.resolve()}", flush=True)
+    result = subprocess.run(command, check=False)
+    if result.returncode != 0 or not args.load:
+        return result.returncode
+    verify = [
+        sys.executable,
+        str(Path(__file__).with_name("verify_crowdb.py")),
+        str(run / "report.json"),
+        "--iceberg-scan",
+        "--require-complete",
+    ]
+    if args.py_io_impl:
+        verify += ["--py-io-impl", args.py_io_impl]
+    if args.full_read:
+        verify += ["--full-read"]
+    return subprocess.run(verify, check=False).returncode
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())