dlt-iceberg 0.1.4__py3-none-any.whl → 0.2.0__py3-none-any.whl

dlt_iceberg-0.1.4.dist-info/METADATA

@@ -1,314 +0,0 @@
-Metadata-Version: 2.4
-Name: dlt-iceberg
-Version: 0.1.4
-Summary: dlt destination for Apache Iceberg with atomic multi-file commits via REST catalogs
-Project-URL: Homepage, https://github.com/sidequery/dlt-iceberg
-Project-URL: Repository, https://github.com/sidequery/dlt-iceberg
-Project-URL: Issues, https://github.com/sidequery/dlt-iceberg/issues
-Author-email: Sidequery <hello@sidequery.com>
-License: MIT
-License-File: LICENSE
-Keywords: data-engineering,data-pipeline,dlt,elt,etl,iceberg
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Database
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.11
-Requires-Dist: boto3>=1.40.50
-Requires-Dist: dlt>=1.17.1
-Requires-Dist: pandas>=2.3.3
-Requires-Dist: pyarrow>=21.0.0
-Requires-Dist: pydantic<2.11
-Requires-Dist: pyiceberg[pyiceberg-core]>=0.10.0
-Requires-Dist: requests>=2.32.5
-Requires-Dist: s3fs>=0.4.2
-Requires-Dist: sqlalchemy>=2.0.44
-Description-Content-Type: text/markdown
-
-# dlt-iceberg
-
-A [dlt](https://dlthub.com/) destination for [Apache Iceberg](https://iceberg.apache.org/) tables using REST catalogs.
-
-## Features
-
-- **Atomic Multi-File Commits**: Multiple parquet files committed as single Iceberg snapshot per table
-- **REST Catalog Support**: Works with Nessie, Polaris, AWS Glue, Unity Catalog
-- **Partitioning**: Full support for Iceberg partition transforms (temporal, bucket, truncate, identity)
-- **Authentication**: OAuth2, Bearer token, AWS SigV4
-- **Write Dispositions**: Append, replace, merge (upsert)
-- **Schema Evolution**: Automatic schema updates when adding columns
-- **Retry Logic**: Exponential backoff for transient failures
-
-## Installation
-
-```bash
-pip install dlt-iceberg
-```
-
-Or with uv:
-
-```bash
-uv add dlt-iceberg
-```
-
-## Quick Start
-
-See [examples/](examples/) directory for working examples.
-
-### Incremental Load
-
-```python
-import dlt
-from dlt_iceberg import iceberg_rest
-
-@dlt.resource(name="events", write_disposition="append")
-def generate_events():
-    yield {"event_id": 1, "value": 100}
-
-pipeline = dlt.pipeline(
-    pipeline_name="my_pipeline",
-    destination=iceberg_rest(
-        catalog_uri="http://localhost:19120/iceberg/main",
-        namespace="analytics",
-        s3_endpoint="http://localhost:9000",
-        s3_access_key_id="minioadmin",
-        s3_secret_access_key="minioadmin",
-        s3_region="us-east-1",
-    ),
-)
-
-pipeline.run(generate_events())
-```
-
-### Merge/Upsert
-
-```python
-@dlt.resource(
-    name="users",
-    write_disposition="merge",
-    primary_key="user_id"
-)
-def generate_users():
-    yield {"user_id": 1, "name": "Alice", "status": "active"}
-
-pipeline.run(generate_users())
-```
-
-## Configuration Options
-
-All configuration options can be passed to `iceberg_rest()`:
-
-```python
-iceberg_rest(
-    catalog_uri="...",           # Required: REST catalog URI
-    namespace="...",             # Required: Iceberg namespace (database)
-    warehouse="...",             # Optional: Warehouse location
-
-    # Authentication
-    credential="...",            # OAuth2 client credentials
-    oauth2_server_uri="...",     # OAuth2 token endpoint
-    token="...",                 # Bearer token
-
-    # AWS SigV4
-    sigv4_enabled=True,
-    signing_region="us-east-1",
-
-    # S3 configuration
-    s3_endpoint="...",
-    s3_access_key_id="...",
-    s3_secret_access_key="...",
-    s3_region="...",
-
-    # Performance tuning
-    max_retries=5,               # Retry attempts for transient failures
-    retry_backoff_base=2.0,      # Exponential backoff multiplier
-    merge_batch_size=100000,     # Rows per batch for merge operations
-    strict_casting=False,        # Fail on potential data loss
-)
-```
-
-### Nessie (Docker)
-
-```python
-iceberg_rest(
-    catalog_uri="http://localhost:19120/iceberg/main",
-    namespace="my_namespace",
-    s3_endpoint="http://localhost:9000",
-    s3_access_key_id="minioadmin",
-    s3_secret_access_key="minioadmin",
-    s3_region="us-east-1",
-)
-```
-
-Start services: `docker compose up -d`
-
-### AWS Glue
-
-```python
-iceberg_rest(
-    catalog_uri="https://glue.us-east-1.amazonaws.com/iceberg",
-    warehouse="<account-id>:s3tablescatalog/<bucket>",
-    namespace="my_database",
-    sigv4_enabled=True,
-    signing_region="us-east-1",
-)
-```
-
-AWS credentials via environment variables.
-
-### Polaris
-
-```python
-iceberg_rest(
-    catalog_uri="https://polaris.example.com/api/catalog",
-    warehouse="s3://bucket/warehouse",
-    namespace="production",
-    credential="client-id:client-secret",
-    oauth2_server_uri="https://polaris.example.com/api/catalog/v1/oauth/tokens",
-)
-```
-
-### Unity Catalog
-
-```python
-iceberg_rest(
-    catalog_uri="https://<workspace>.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest",
-    warehouse="<catalog-name>",
-    namespace="<schema-name>",
-    token="<databricks-token>",
-)
-```
-
-## Partitioning
-
-Mark columns for partitioning using dlt column hints:
-
-```python
-@dlt.resource(
-    name="events",
-    columns={
-        "event_date": {
-            "data_type": "date",
-            "partition": True,
-            "partition_transform": "day",  # Optional: year, month, day, hour
-        },
-        "region": {
-            "data_type": "text",
-            "partition": True,  # Uses identity transform for strings
-        },
-        "user_id": {
-            "data_type": "bigint",
-            "partition": True,
-            "partition_transform": "bucket[10]",  # Hash into 10 buckets
-        }
-    }
-)
-def events():
-    ...
-```
-
-### Available Transforms
-
-- **Temporal**: `year`, `month`, `day`, `hour` (for timestamp/date columns)
-- **Identity**: No transformation (default for string/integer)
-- **Bucket**: `bucket[N]` - Hash-based partitioning into N buckets
-- **Truncate**: `truncate[N]` - Truncate strings/integers to N width
-
-### Default Behavior
-
-If `partition_transform` is not specified:
-- Timestamp/date columns default to `month`
-- String/integer columns default to `identity`
-
-## Write Dispositions
-
-### Append
-```python
-write_disposition="append"
-```
-Adds new data without modifying existing rows.
-
-### Replace
-```python
-write_disposition="replace"
-```
-Truncates table and inserts new data.
-
-### Merge
-```python
-write_disposition="merge"
-primary_key="user_id"
-```
-Updates existing rows by primary key, inserts new rows.
-
-## Development
-
-### Run Tests
-
-```bash
-# Start Docker services
-docker compose up -d
-
-# Run all tests
-uv run pytest tests/ -v
-
-# Run only unit tests
-uv run pytest tests/ -v -m "not integration"
-
-# Run only integration tests
-uv run pytest tests/ -v -m integration
-```
-
-### Project Structure
-
-```
-dlt-iceberg/
-├── src/dlt_iceberg/
-│   ├── __init__.py              # Public API
-│   ├── destination_client.py   # Class-based destination (atomic commits)
-│   ├── destination.py           # Function-based destination (legacy)
-│   ├── schema_converter.py     # dlt → Iceberg schema conversion
-│   ├── schema_casting.py        # Arrow table casting
-│   ├── schema_evolution.py     # Schema updates
-│   ├── partition_builder.py    # Partition specs
-│   └── error_handling.py       # Retry logic
-├── tests/
-│   ├── test_destination_rest_catalog.py  # Integration tests (Docker)
-│   ├── test_class_based_atomic.py        # Atomic commit tests
-│   ├── test_merge_disposition.py
-│   ├── test_schema_evolution.py
-│   └── ...
-├── examples/
-│   ├── incremental_load.py     # CSV incremental loading
-│   ├── merge_load.py            # CSV merge/upsert
-│   └── data/                    # Sample CSV files
-└── docker-compose.yml           # Nessie + MinIO for testing
-```
-
-## How It Works
-
-The class-based destination uses dlt's `JobClientBase` interface to accumulate parquet files during a load and commit them atomically in `complete_load()`:
-
-1. dlt extracts data and writes parquet files
-2. Each file is registered in module-level global state
-3. After all files complete, `complete_load()` is called
-4. All files for a table are combined and committed as single Iceberg snapshot
-5. Each table gets one snapshot per load
-
-This ensures atomic commits even though dlt creates multiple client instances.
-
-## License
-
-MIT License - see LICENSE file
-
-## Resources
-
-- [dlt Documentation](https://dlthub.com/docs)
-- [Apache Iceberg](https://iceberg.apache.org/)
-- [PyIceberg](https://py.iceberg.apache.org/)
-- [Iceberg REST Spec](https://iceberg.apache.org/rest-catalog-spec/)

dlt_iceberg-0.1.4.dist-info/RECORD

@@ -1,12 +0,0 @@
-dlt_iceberg/__init__.py,sha256=ONy6E-sGcCvvqia8_fGaYp8da4n4wdjox9W42tmQPK0,780
-dlt_iceberg/destination.py,sha256=F8QJXsQeosOA32Xm1140DL485WQmxbuhiA2QZ6zpVSU,15737
-dlt_iceberg/destination_client.py,sha256=l1q8GYvIJ_tBgoQ979IS3VtUQNmg2-hYv80XZkAVFKs,23786
-dlt_iceberg/error_handling.py,sha256=k6Kkldi9BDRsXQ63VEBMMSw1xx2-b1BMjsgRFKI2iB0,7852
-dlt_iceberg/partition_builder.py,sha256=l9YNAh2t6gk2xqsPSOs8ymTDLk9BOEZWVOtVni7ONNU,10081
-dlt_iceberg/schema_casting.py,sha256=oSQrnOcCMFcinMS65N8YQ1uzrqnQmN50mCCuQyE3794,15247
-dlt_iceberg/schema_converter.py,sha256=e_eqXQz2cpABOGEAxVwcGbiOdVmv9kaZanRnU83lzXk,5619
-dlt_iceberg/schema_evolution.py,sha256=ieOkCA9ngQdJ5lbZLYQ09deTLZEW8whxDn2arpoH-aM,8326
-dlt_iceberg-0.1.4.dist-info/METADATA,sha256=2FnlwL3bwN-aItvx0Y1dUQ3iTfjLr9AJUQ0hLrJZql8,8777
-dlt_iceberg-0.1.4.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-dlt_iceberg-0.1.4.dist-info/licenses/LICENSE,sha256=0amGlcH0msYju3WUhlsuUxO4aj3ZODkkIZ0MKOq9fQ4,1066
-dlt_iceberg-0.1.4.dist-info/RECORD,,