dlt-iceberg 0.1.2__tar.gz → 0.1.4__tar.gz

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dlt-iceberg
-Version: 0.1.2
+Version: 0.1.4
 Summary: dlt destination for Apache Iceberg with atomic multi-file commits via REST catalogs
 Project-URL: Homepage, https://github.com/sidequery/dlt-iceberg
 Project-URL: Repository, https://github.com/sidequery/dlt-iceberg
@@ -47,9 +47,13 @@ A [dlt](https://dlthub.com/) destination for [Apache Iceberg](https://iceberg.ap
 ## Installation
 
 ```bash
-git clone https://github.com/sidequery/dlt-iceberg.git
-cd dlt-iceberg
-uv sync
+pip install dlt-iceberg
+```
+
+Or with uv:
+
+```bash
+uv add dlt-iceberg
 ```
 
 ## Quick Start
@@ -95,7 +99,38 @@ def generate_users():
 pipeline.run(generate_users())
 ```
 
-## Configuration
+## Configuration Options
+
+All configuration options can be passed to `iceberg_rest()`:
+
+```python
+iceberg_rest(
+    catalog_uri="...",           # Required: REST catalog URI
+    namespace="...",             # Required: Iceberg namespace (database)
+    warehouse="...",             # Optional: Warehouse location
+
+    # Authentication
+    credential="...",            # OAuth2 client credentials
+    oauth2_server_uri="...",     # OAuth2 token endpoint
+    token="...",                 # Bearer token
+
+    # AWS SigV4
+    sigv4_enabled=True,
+    signing_region="us-east-1",
+
+    # S3 configuration
+    s3_endpoint="...",
+    s3_access_key_id="...",
+    s3_secret_access_key="...",
+    s3_region="...",
+
+    # Performance tuning
+    max_retries=5,               # Retry attempts for transient failures
+    retry_backoff_base=2.0,      # Exponential backoff multiplier
+    merge_batch_size=100000,     # Rows per batch for merge operations
+    strict_casting=False,        # Fail on potential data loss
+)
+```
 
 ### Nessie (Docker)
 

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/README.md

@@ -15,9 +15,13 @@ A [dlt](https://dlthub.com/) destination for [Apache Iceberg](https://iceberg.ap
 ## Installation
 
 ```bash
-git clone https://github.com/sidequery/dlt-iceberg.git
-cd dlt-iceberg
-uv sync
+pip install dlt-iceberg
+```
+
+Or with uv:
+
+```bash
+uv add dlt-iceberg
 ```
 
 ## Quick Start
@@ -63,7 +67,38 @@ def generate_users():
 pipeline.run(generate_users())
 ```
 
-## Configuration
+## Configuration Options
+
+All configuration options can be passed to `iceberg_rest()`:
+
+```python
+iceberg_rest(
+    catalog_uri="...",           # Required: REST catalog URI
+    namespace="...",             # Required: Iceberg namespace (database)
+    warehouse="...",             # Optional: Warehouse location
+
+    # Authentication
+    credential="...",            # OAuth2 client credentials
+    oauth2_server_uri="...",     # OAuth2 token endpoint
+    token="...",                 # Bearer token
+
+    # AWS SigV4
+    sigv4_enabled=True,
+    signing_region="us-east-1",
+
+    # S3 configuration
+    s3_endpoint="...",
+    s3_access_key_id="...",
+    s3_secret_access_key="...",
+    s3_region="...",
+
+    # Performance tuning
+    max_retries=5,               # Retry attempts for transient failures
+    retry_backoff_base=2.0,      # Exponential backoff multiplier
+    merge_batch_size=100000,     # Rows per batch for merge operations
+    strict_casting=False,        # Fail on potential data loss
+)
+```
 
 ### Nessie (Docker)
 

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/examples/README.md

@@ -2,6 +2,8 @@
 
 This directory contains example scripts demonstrating how to use dlt-iceberg with REST catalogs.
 
+All examples use [PEP 723](https://peps.python.org/pep-0723/) inline script metadata, allowing them to be run directly with `uv run` without installing dependencies separately.
+
 ## Prerequisites
 
 Start the Docker services (Nessie REST catalog and MinIO):
@@ -48,6 +50,32 @@ Demonstrates merging two CSV files with overlapping customer IDs using merge dis
 uv run examples/merge_load.py
 ```
 
+### USGS Earthquake Data (`usgs_earthquakes.py`)
+
+Demonstrates loading real-world GeoJSON data from the USGS Earthquake API from 2010 through current date.
+
+- Fetches earthquake data from USGS API (2010-present, ~190 months)
+- Loads data in monthly batches with automatic weekly splitting for high-volume months
+- Handles API's 20,000 result limit by automatically splitting large months into weekly chunks
+- Uses partitioning by month on timestamp column
+- Transforms GeoJSON features into flat records
+- Loads ~2.5 million earthquakes with complete metadata
+
+**Features demonstrated:**
+- Dynamic date range generation using `date.today()`
+- Automatic handling of API result limits with recursive splitting
+- Retry logic with exponential backoff
+- Rate limiting between API requests
+- Large-scale data ingestion (190+ API calls)
+
+**Run:**
+
+```bash
+uv run examples/usgs_earthquakes.py
+```
+
+**Note:** This script takes approximately 15-20 minutes to complete as it makes 190+ API calls with rate limiting.
+
 ## Data Files
 
 Sample CSV files are in the `data/` directory:
@@ -63,4 +91,6 @@ Sample CSV files are in the `data/` directory:
 - **Incremental loads**: Append new data to existing tables
 - **Merge/Upsert**: Update existing records and insert new ones based on primary key
 - **REST catalog**: All examples use Nessie REST catalog with MinIO storage
+- **Partitioning**: Partition tables by timestamp (month transform)
+- **API integration**: Fetch and transform data from external APIs
 - **Querying**: Direct PyIceberg queries to verify loaded data

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/examples/incremental_load.py

@@ -1,3 +1,11 @@
+#!/usr/bin/env -S uv run
+# /// script
+# dependencies = [
+#     "dlt",
+#     "dlt-iceberg",
+#     "pyiceberg",
+# ]
+# ///
 """
 Incremental Load Example
 

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/examples/merge_load.py

@@ -1,3 +1,12 @@
+#!/usr/bin/env -S uv run
+# /// script
+# dependencies = [
+#     "dlt",
+#     "dlt-iceberg",
+#     "pyiceberg",
+#     "pandas",
+# ]
+# ///
 """
 Merge Load Example
 

dlt_iceberg-0.1.4/examples/usgs_earthquakes.py

@@ -0,0 +1,234 @@
+#!/usr/bin/env -S uv run
+# /// script
+# dependencies = [
+#     "dlt",
+#     "dlt-iceberg",
+#     "pyiceberg",
+#     "requests",
+#     "python-dateutil",
+# ]
+# ///
+"""
+USGS Earthquake Data Example
+
+Loads earthquake data from USGS GeoJSON API from 2010 through current date
+into an Iceberg table.
+"""
+
+import dlt
+import requests
+import time
+from datetime import datetime, date
+from dateutil.relativedelta import relativedelta
+from dlt_iceberg import iceberg_rest
+
+
+def fetch_earthquakes(start_date: str, end_date: str, split_on_error: bool = True):
+    """
+    Fetch earthquake data from USGS API for a date range.
+
+    If a 400 error occurs (likely due to >20k result limit), automatically
+    splits the range into smaller chunks and retries.
+
+    Args:
+        start_date: Start date in YYYY-MM-DD format
+        end_date: End date in YYYY-MM-DD format
+        split_on_error: If True, split the date range on 400 errors
+    """
+    url = "https://earthquake.usgs.gov/fdsnws/event/1/query"
+    params = {
+        "format": "geojson",
+        "starttime": start_date,
+        "endtime": end_date,
+    }
+
+    print(f"Fetching earthquakes from {start_date} to {end_date}...")
+
+    # Retry logic for rate limiting
+    max_retries = 3
+    for attempt in range(max_retries):
+        try:
+            response = requests.get(url, params=params, timeout=30)
+            response.raise_for_status()
+            break
+        except requests.exceptions.HTTPError as e:
+            if e.response.status_code == 400 and split_on_error:
+                # Split the date range and retry with smaller chunks
+                print(f"  Too many results, splitting range into weekly chunks...")
+                from datetime import timedelta
+
+                start_dt = datetime.fromisoformat(start_date)
+                end_dt = datetime.fromisoformat(end_date)
+                current = start_dt
+
+                while current < end_dt:
+                    next_week = min(current + timedelta(days=7), end_dt)
+                    # Recursively fetch with split_on_error=False to avoid infinite recursion
+                    yield from fetch_earthquakes(
+                        current.date().isoformat(),
+                        next_week.date().isoformat(),
+                        split_on_error=False
+                    )
+                    current = next_week
+                return
+            elif e.response.status_code == 400:
+                print(f"  Warning: API returned 400 error for {start_date} to {end_date}, skipping...")
+                return
+            if attempt < max_retries - 1:
+                wait_time = 2 ** attempt
+                print(f"  Request failed, retrying in {wait_time}s...")
+                time.sleep(wait_time)
+            else:
+                raise
+        except requests.exceptions.RequestException as e:
+            if attempt < max_retries - 1:
+                wait_time = 2 ** attempt
+                print(f"  Request failed, retrying in {wait_time}s...")
+                time.sleep(wait_time)
+            else:
+                raise
+
+    data = response.json()
+    features = data.get("features", [])
+    print(f"Retrieved {len(features)} earthquakes")
+
+    # Rate limiting: sleep briefly between requests
+    time.sleep(0.5)
+
+    # Transform GeoJSON features into flat records
+    for feature in features:
+        props = feature["properties"]
+        geom = feature["geometry"]
+
+        yield {
+            "earthquake_id": feature["id"],
+            "magnitude": props.get("mag"),
+            "place": props.get("place"),
+            "time": datetime.fromtimestamp(props["time"] / 1000) if props.get("time") else None,
+            "updated": datetime.fromtimestamp(props["updated"] / 1000) if props.get("updated") else None,
+            "url": props.get("url"),
+            "detail": props.get("detail"),
+            "felt": props.get("felt"),
+            "cdi": props.get("cdi"),
+            "mmi": props.get("mmi"),
+            "alert": props.get("alert"),
+            "status": props.get("status"),
+            "tsunami": props.get("tsunami"),
+            "sig": props.get("sig"),
+            "net": props.get("net"),
+            "code": props.get("code"),
+            "ids": props.get("ids"),
+            "sources": props.get("sources"),
+            "types": props.get("types"),
+            "nst": props.get("nst"),
+            "dmin": props.get("dmin"),
+            "rms": props.get("rms"),
+            "gap": props.get("gap"),
+            "magType": props.get("magType"),
+            "type": props.get("type"),
+            "title": props.get("title"),
+            "longitude": geom["coordinates"][0] if geom and geom.get("coordinates") else None,
+            "latitude": geom["coordinates"][1] if geom and geom.get("coordinates") else None,
+            "depth": geom["coordinates"][2] if geom and geom.get("coordinates") and len(geom["coordinates"]) > 2 else None,
+        }
+
+
+def main():
+    # Create dlt pipeline with Nessie REST catalog
+    pipeline = dlt.pipeline(
+        pipeline_name="usgs_earthquakes",
+        destination=iceberg_rest(
+            catalog_uri="http://localhost:19120/iceberg/main",
+            namespace="examples",
+            s3_endpoint="http://localhost:9000",
+            s3_access_key_id="minioadmin",
+            s3_secret_access_key="minioadmin",
+            s3_region="us-east-1",
+        ),
+        dataset_name="usgs_data",
+    )
+
+    # Load earthquakes from 2010 through current date
+    # Breaking into monthly batches to avoid overwhelming the API
+    # Note: USGS endtime is exclusive, so we use the first day of next month
+
+    start_date = date(2010, 1, 1)
+    end_date = date.today()
+
+    date_ranges = []
+    current = start_date
+    while current <= end_date:
+        next_month = current + relativedelta(months=1)
+        date_ranges.append((current.isoformat(), next_month.isoformat()))
+        current = next_month
+
+    print(f"Loading {len(date_ranges)} months of earthquake data from {start_date} to {end_date}...")
+    print()
+
+    for i, (start, end) in enumerate(date_ranges, 1):
+        @dlt.resource(
+            name="earthquakes",
+            write_disposition="append",
+            columns={
+                "time": {
+                    "data_type": "timestamp",
+                    "x-partition": True,
+                    "x-partition-transform": "month",
+                }
+            }
+        )
+        def earthquakes_batch():
+            return fetch_earthquakes(start, end)
+
+        load_info = pipeline.run(earthquakes_batch())
+        print(f"[{i}/{len(date_ranges)}] Loaded {start} to {end}")
+        print()
+
+    # Query the table to verify
+    from pyiceberg.catalog import load_catalog
+
+    catalog = load_catalog(
+        "query",
+        type="rest",
+        uri="http://localhost:19120/iceberg/main",
+        **{
+            "s3.endpoint": "http://localhost:9000",
+            "s3.access-key-id": "minioadmin",
+            "s3.secret-access-key": "minioadmin",
+            "s3.region": "us-east-1",
+        },
+    )
+
+    table = catalog.load_table("examples.earthquakes")
+    result = table.scan().to_arrow()
+
+    print(f"\n{'='*60}")
+    print(f"Total earthquakes loaded: {len(result)}")
+
+    import pyarrow.compute as pc
+    print(f"Date range: {pc.min(result['time']).as_py()} to {pc.max(result['time']).as_py()}")
+    print(f"Magnitude range: {pc.min(result['magnitude']).as_py()} to {pc.max(result['magnitude']).as_py()}")
+
+    # Show some sample records
+    import pandas as pd
+    df = result.to_pandas()
+    print(f"\nSample earthquakes:")
+    print(df[["time", "magnitude", "place", "depth"]].head(10).to_string(index=False))
+
+    # Show distribution by month
+    df["month"] = pd.to_datetime(df["time"]).dt.to_period("M")
+    monthly_counts = df.groupby("month").size().sort_index()
+    print(f"\nEarthquakes by month:")
+    for month, count in monthly_counts.items():
+        print(f"  {month}: {count:,}")
+
+    # Show magnitude distribution
+    print(f"\nMagnitude distribution:")
+    print(df["magnitude"].describe())
+
+    print(f"\n{'='*60}")
+    print("USGS earthquake data load complete!")
+
+
+if __name__ == "__main__":
+    main()

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dlt-iceberg"
-version = "0.1.2"
+version = "0.1.4"
 description = "dlt destination for Apache Iceberg with atomic multi-file commits via REST catalogs"
 readme = "README.md"
 requires-python = ">=3.11"

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/src/dlt_iceberg/destination_client.py

@@ -34,7 +34,11 @@ from pyiceberg.exceptions import (
 from .schema_converter import convert_dlt_to_iceberg_schema
 from .partition_builder import build_partition_spec
 from .schema_evolution import evolve_schema_if_needed, SchemaEvolutionError
-from .schema_casting import cast_table_safe, CastingError
+from .schema_casting import (
+    cast_table_safe,
+    CastingError,
+    ensure_iceberg_compatible_arrow_data,
+)
 from .error_handling import (
     is_retryable_error,
     log_error_with_context,
@@ -89,6 +93,9 @@ class IcebergRestConfiguration(DestinationClientConfiguration):
     # Schema casting configuration
     strict_casting: bool = False
 
+    # Merge batch size (for upsert operations to avoid memory issues)
+    merge_batch_size: int = 100000
+
 
 class IcebergRestLoadJob(RunnableLoadJob):
     """
@@ -380,7 +387,8 @@ class IcebergRestClient(JobClientBase):
                 # Create table if needed
                 if not table_exists:
                     # Use first file's Arrow table to generate schema
-                    first_arrow_table = file_data[0][2]
+                    # Apply Iceberg compatibility first so schema uses compatible types
+                    first_arrow_table = ensure_iceberg_compatible_arrow_data(file_data[0][2])
                     iceberg_schema = convert_dlt_to_iceberg_schema(
                         table_schema, first_arrow_table
                     )
@@ -401,7 +409,7 @@ class IcebergRestClient(JobClientBase):
                     logger.info(f"Created table {identifier} at {iceberg_table.location()}")
                 else:
                     # Table exists - check if schema evolution is needed
-                    first_arrow_table = file_data[0][2]
+                    first_arrow_table = ensure_iceberg_compatible_arrow_data(file_data[0][2])
                     incoming_schema = convert_dlt_to_iceberg_schema(
                         table_schema, first_arrow_table
                     )
@@ -415,12 +423,15 @@ class IcebergRestClient(JobClientBase):
                         logger.info(f"Schema evolved for table {identifier}")
                         iceberg_table = catalog.load_table(identifier)
 
-                # Combine all Arrow tables and cast to match Iceberg schema
+                # Get expected schema (already has Iceberg-compatible types from creation)
                 expected_schema = schema_to_pyarrow(iceberg_table.schema())
+
+                # Combine all Arrow tables and cast to match Iceberg schema
                 combined_tables = []
 
                 for _, file_path, arrow_table in file_data:
-                    # Cast each table to match Iceberg schema
+                    # Cast to match Iceberg schema
+                    # (compatibility conversions already applied when schema was created)
                     casted_table = cast_table_safe(
                         arrow_table,
                         expected_schema,
@@ -463,15 +474,34 @@ class IcebergRestClient(JobClientBase):
                         iceberg_table.append(combined_table)
                     else:
                         logger.info(f"Merging into table {identifier} on keys {primary_keys}")
-                        upsert_result = iceberg_table.upsert(
-                            df=combined_table,
-                            join_cols=primary_keys,
-                            when_matched_update_all=True,
-                            when_not_matched_insert_all=True,
-                        )
+
+                        # Batch upserts to avoid memory issues on large datasets
+                        batch_size = self.config.merge_batch_size
+                        total_updated = 0
+                        total_inserted = 0
+
+                        for batch_start in range(0, len(combined_table), batch_size):
+                            batch_end = min(batch_start + batch_size, len(combined_table))
+                            batch = combined_table.slice(batch_start, batch_end - batch_start)
+
+                            logger.info(
+                                f"Upserting batch {batch_start//batch_size + 1}: "
+                                f"rows {batch_start} to {batch_end} ({len(batch)} rows)"
+                            )
+
+                            upsert_result = iceberg_table.upsert(
+                                df=batch,
+                                join_cols=primary_keys,
+                                when_matched_update_all=True,
+                                when_not_matched_insert_all=True,
+                            )
+
+                            total_updated += upsert_result.rows_updated
+                            total_inserted += upsert_result.rows_inserted
+
                         logger.info(
-                            f"Upsert completed: {upsert_result.rows_updated} updated, "
-                            f"{upsert_result.rows_inserted} inserted"
+                            f"Upsert completed: {total_updated} updated, "
+                            f"{total_inserted} inserted across {(total_rows + batch_size - 1) // batch_size} batches"
                         )
                 else:
                     raise ValueError(f"Unknown write disposition: {write_disposition}")

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/src/dlt_iceberg/schema_casting.py

@@ -6,12 +6,75 @@ and allow users to control casting behavior.
 """
 
 import logging
-from typing import List, Optional, Tuple
+from typing import List, Optional, Tuple, Dict, Callable
 import pyarrow as pa
 
 logger = logging.getLogger(__name__)
 
 
+def ensure_iceberg_compatible_arrow_schema(schema: pa.Schema) -> pa.Schema:
+    """
+    Convert Arrow schema to Iceberg-compatible schema.
+
+    Converts types that Iceberg doesn't support:
+    - time32 → time64 (microseconds)
+    - decimal256 → string (Iceberg only supports decimal128)
+    - dictionary → value_type (unwrap dictionary encoding)
+
+    Args:
+        schema: PyArrow schema
+
+    Returns:
+        Iceberg-compatible PyArrow schema
+    """
+    def convert_field(field: pa.Field) -> pa.Field:
+        field_type = field.type
+
+        # time32 → time64(us)
+        if pa.types.is_time32(field_type):
+            return pa.field(field.name, pa.time64("us"), nullable=field.nullable)
+
+        # decimal256 → string (pyarrow doesn't allow downcasting to decimal128)
+        if pa.types.is_decimal256(field_type):
+            logger.warning(
+                f"Converting decimal256 field '{field.name}' to string "
+                f"(Iceberg doesn't support decimal256)"
+            )
+            return pa.field(field.name, pa.string(), nullable=field.nullable)
+
+        # dictionary → value_type (unwrap dictionary encoding)
+        if pa.types.is_dictionary(field_type):
+            return pa.field(field.name, field_type.value_type, nullable=field.nullable)
+
+        # list/struct types - recursively convert nested fields
+        if pa.types.is_list(field_type):
+            value_field = convert_field(pa.field("item", field_type.value_type))
+            return pa.field(field.name, pa.list_(value_field.type), nullable=field.nullable)
+
+        if pa.types.is_struct(field_type):
+            new_fields = [convert_field(f) for f in field_type]
+            return pa.field(field.name, pa.struct(new_fields), nullable=field.nullable)
+
+        return field
+
+    new_fields = [convert_field(field) for field in schema]
+    return pa.schema(new_fields)
+
+
+def ensure_iceberg_compatible_arrow_data(table: pa.Table) -> pa.Table:
+    """
+    Convert Arrow table to Iceberg-compatible schema and cast data.
+
+    Args:
+        table: PyArrow table
+
+    Returns:
+        Table with Iceberg-compatible schema
+    """
+    new_schema = ensure_iceberg_compatible_arrow_schema(table.schema)
+    return table.cast(new_schema)
+
+
 class CastingError(Exception):
     """Raised when a cast would result in data loss in strict mode."""
     pass

{dlt_iceberg-0.1.2 → dlt_iceberg-0.1.4}/uv.lock

@@ -182,7 +182,7 @@ wheels = [
 
 [[package]]
 name = "dlt-iceberg"
-version = "0.1.1"
+version = "0.1.3"
 source = { editable = "." }
 dependencies = [
     { name = "boto3" },