ducklake-delta-exporter 0.1.0__tar.gz → 0.1.2__tar.gz

ducklake_delta_exporter-0.1.2/PKG-INFO

@@ -0,0 +1,72 @@
+Metadata-Version: 2.4
+Name: ducklake-delta-exporter
+Version: 0.1.2
+Summary: A utility to export DuckLake database metadata to Delta Lake transaction logs.
+Home-page: https://github.com/djouallah/ducklake_delta_exporter
+Author: mim
+Author-email: your.email@example.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Development Status :: 3 - Alpha
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: duckdb
+Requires-Dist: pyarrow
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🦆 DuckLake Delta Exporter
+
+A Python utility to **bridge the gap between DuckLake and Delta Lake** by generating Delta-compatible transaction logs directly from DuckLake metadata.
+
+This isn’t your typical general-purpose library. It’s mostly battle-tested with **OneLake mounted storage**, and while it *should* work with local filesystems, there’s **no support for S3, GCS, or ABFSS** .
+
+It doesn’t use the `deltalake` Python package either. The metadata is handcrafted from scratch — because why not reinvent the wheel for fun and learning?
+
+**Goal?**  
+Mostly to annoy DuckDB developers into finally shipping a proper Delta Lake metadata exporter  😎
+
+🔗 [Source code on GitHub](https://github.com/djouallah/ducklake_delta_exporter)
+
+---
+
+## ✨ Features
+
+- **DuckLake → Delta Sync**  
+  Generates Delta Lake `_delta_log/*.json` transaction files and Parquet checkpoints from the latest DuckLake state.
+
+- **Schema Mapping**  
+  Converts DuckDB types to their Spark SQL equivalents so Delta can understand them without throwing a tantrum.
+
+- **Change Detection**  
+  Detects file-level additions/removals since the last export — keeps things incremental and tidy.
+
+- **Checkpointing**  
+  Automatically writes Delta checkpoints every N versions (configurable), so readers don’t have to replay the entire log from scratch.
+
+---
+
+## ⚙️ Installation & Usage
+
+Install via pip:
+
+```bash
+pip install ducklake-delta-exporter
+```
+
+```
+from ducklake_delta_exporter import generate_latest_delta_log
+
+generate_latest_delta_log('/lakehouse/default/Files/meta.db','/lakehouse/default/Tables')
+```
+the data path is optional, but handy to support relative path

ducklake_delta_exporter-0.1.2/README.md

@@ -0,0 +1,45 @@
+# 🦆 DuckLake Delta Exporter
+
+A Python utility to **bridge the gap between DuckLake and Delta Lake** by generating Delta-compatible transaction logs directly from DuckLake metadata.
+
+This isn’t your typical general-purpose library. It’s mostly battle-tested with **OneLake mounted storage**, and while it *should* work with local filesystems, there’s **no support for S3, GCS, or ABFSS** .
+
+It doesn’t use the `deltalake` Python package either. The metadata is handcrafted from scratch — because why not reinvent the wheel for fun and learning?
+
+**Goal?**  
+Mostly to annoy DuckDB developers into finally shipping a proper Delta Lake metadata exporter  😎
+
+🔗 [Source code on GitHub](https://github.com/djouallah/ducklake_delta_exporter)
+
+---
+
+## ✨ Features
+
+- **DuckLake → Delta Sync**  
+  Generates Delta Lake `_delta_log/*.json` transaction files and Parquet checkpoints from the latest DuckLake state.
+
+- **Schema Mapping**  
+  Converts DuckDB types to their Spark SQL equivalents so Delta can understand them without throwing a tantrum.
+
+- **Change Detection**  
+  Detects file-level additions/removals since the last export — keeps things incremental and tidy.
+
+- **Checkpointing**  
+  Automatically writes Delta checkpoints every N versions (configurable), so readers don’t have to replay the entire log from scratch.
+
+---
+
+## ⚙️ Installation & Usage
+
+Install via pip:
+
+```bash
+pip install ducklake-delta-exporter
+```
+
+```
+from ducklake_delta_exporter import generate_latest_delta_log
+
+generate_latest_delta_log('/lakehouse/default/Files/meta.db','/lakehouse/default/Tables')
+```
+the data path is optional, but handy to support relative path

{ducklake_delta_exporter-0.1.0 → ducklake_delta_exporter-0.1.2}/ducklake_delta_exporter/__init__.py

@@ -160,7 +160,7 @@ def get_latest_delta_version_info(delta_log_path, con, table_id):
     return last_delta_version_idx, files_in_last_delta_version, last_exported_ducklake_snapshot_id, meta_id_from_delta_log
 
 
-def generate_latest_delta_log(db_path: str, data_root: str='/lakehouse/default/Tables', checkpoint_interval: int = 1):
+def generate_latest_delta_log(db_path: str, data_root: str = None):
     """
     Generates a Delta Lake transaction log for the LATEST state of each table in a DuckLake database.
     This creates incremental updates to Delta, not a full history.
@@ -168,10 +168,11 @@ def generate_latest_delta_log(db_path: str, data_root: str='/lakehouse/default/T
     Args:
         db_path (str): The path to the DuckLake database file.
         data_root (str): The root directory for the lakehouse data.
-        checkpoint_interval (int): The interval at which to create checkpoint files.
     """
+    checkpoint_interval = 1
     con = duckdb.connect(db_path, read_only=True)
-
+    if data_root is None: # Only fetch from DB if not provided by user
+     data_root = con.sql(""" SELECT value FROM ducklake_metadata WHERE key = 'data_path' """).fetchone()[0]
     tables = con.sql("""
         SELECT 
             t.table_id, 

ducklake_delta_exporter-0.1.2/ducklake_delta_exporter.egg-info/PKG-INFO

@@ -0,0 +1,72 @@
+Metadata-Version: 2.4
+Name: ducklake-delta-exporter
+Version: 0.1.2
+Summary: A utility to export DuckLake database metadata to Delta Lake transaction logs.
+Home-page: https://github.com/djouallah/ducklake_delta_exporter
+Author: mim
+Author-email: your.email@example.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Development Status :: 3 - Alpha
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: duckdb
+Requires-Dist: pyarrow
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🦆 DuckLake Delta Exporter
+
+A Python utility to **bridge the gap between DuckLake and Delta Lake** by generating Delta-compatible transaction logs directly from DuckLake metadata.
+
+This isn’t your typical general-purpose library. It’s mostly battle-tested with **OneLake mounted storage**, and while it *should* work with local filesystems, there’s **no support for S3, GCS, or ABFSS** .
+
+It doesn’t use the `deltalake` Python package either. The metadata is handcrafted from scratch — because why not reinvent the wheel for fun and learning?
+
+**Goal?**  
+Mostly to annoy DuckDB developers into finally shipping a proper Delta Lake metadata exporter  😎
+
+🔗 [Source code on GitHub](https://github.com/djouallah/ducklake_delta_exporter)
+
+---
+
+## ✨ Features
+
+- **DuckLake → Delta Sync**  
+  Generates Delta Lake `_delta_log/*.json` transaction files and Parquet checkpoints from the latest DuckLake state.
+
+- **Schema Mapping**  
+  Converts DuckDB types to their Spark SQL equivalents so Delta can understand them without throwing a tantrum.
+
+- **Change Detection**  
+  Detects file-level additions/removals since the last export — keeps things incremental and tidy.
+
+- **Checkpointing**  
+  Automatically writes Delta checkpoints every N versions (configurable), so readers don’t have to replay the entire log from scratch.
+
+---
+
+## ⚙️ Installation & Usage
+
+Install via pip:
+
+```bash
+pip install ducklake-delta-exporter
+```
+
+```
+from ducklake_delta_exporter import generate_latest_delta_log
+
+generate_latest_delta_log('/lakehouse/default/Files/meta.db','/lakehouse/default/Tables')
+```
+the data path is optional, but handy to support relative path

{ducklake_delta_exporter-0.1.0 → ducklake_delta_exporter-0.1.2}/setup.py

@@ -3,7 +3,7 @@ from setuptools import setup, find_packages
 
 setup(
     name='ducklake-delta-exporter',
-    version='0.1.0',
+    version='0.1.2',
     packages=find_packages(),
     install_requires=[
         'duckdb',
@@ -12,9 +12,9 @@ setup(
     author='mim',
     author_email='your.email@example.com',
     description='A utility to export DuckLake database metadata to Delta Lake transaction logs.',
-    long_description=open('README.md').read(),
+    long_description=open('README.md', encoding='utf-8').read(),
     long_description_content_type='text/markdown',
-    url='https://github.com/djouallah/ducklake-delta-exporter',
+    url='https://github.com/djouallah/ducklake_delta_exporter',
     classifiers=[
         'Programming Language :: Python :: 3',
         'License :: OSI Approved :: MIT License',

ducklake_delta_exporter-0.1.0/PKG-INFO

@@ -1,52 +0,0 @@
-Metadata-Version: 2.4
-Name: ducklake-delta-exporter
-Version: 0.1.0
-Summary: A utility to export DuckLake database metadata to Delta Lake transaction logs.
-Home-page: https://github.com/djouallah/ducklake-delta-exporter
-Author: mim
-Author-email: your.email@example.com
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Intended Audience :: Developers
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Classifier: Development Status :: 3 - Alpha
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: duckdb
-Requires-Dist: pyarrow
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-
-# DuckLake Delta Exporter
-A Python utility to synchronize metadata from a DuckLake database with Delta Lake transaction logs. This allows you to manage data in DuckLake and make it discoverable and queryable by Delta Lake compatible tools (e.g., Spark, Delta Lake Rust/Python clients).
-
-# Features
-DuckLake to Delta Sync: Generates incremental Delta Lake transaction logs (_delta_log/*.json) and checkpoint files (_delta_log/*.checkpoint.parquet) based on the latest state of tables in a DuckLake database.
-
-Schema Mapping: Automatically maps DuckDB data types to their Spark SQL equivalents for Delta Lake schema definitions.
-
-Change Detection: Identifies added and removed data files since the last Delta export, ensuring only necessary updates are written to the log.
-
-Checkpointing: Supports creating Delta Lake checkpoint files at a configurable interval for efficient state reconstruction.
-
-# Installation
-You can install this package using pip:
-
-pip install ducklake-delta-exporter
-
-
-
-# Usage
-```
-from ducklake_delta_exporter import generate_latest_delta_log
-generate_latest_delta_log('path/to/your/ducklake.db', data_root='/lakehouse/default/Tables', checkpoint_interval=1)
-```

ducklake_delta_exporter-0.1.0/README.md

@@ -1,25 +0,0 @@
-
-# DuckLake Delta Exporter
-A Python utility to synchronize metadata from a DuckLake database with Delta Lake transaction logs. This allows you to manage data in DuckLake and make it discoverable and queryable by Delta Lake compatible tools (e.g., Spark, Delta Lake Rust/Python clients).
-
-# Features
-DuckLake to Delta Sync: Generates incremental Delta Lake transaction logs (_delta_log/*.json) and checkpoint files (_delta_log/*.checkpoint.parquet) based on the latest state of tables in a DuckLake database.
-
-Schema Mapping: Automatically maps DuckDB data types to their Spark SQL equivalents for Delta Lake schema definitions.
-
-Change Detection: Identifies added and removed data files since the last Delta export, ensuring only necessary updates are written to the log.
-
-Checkpointing: Supports creating Delta Lake checkpoint files at a configurable interval for efficient state reconstruction.
-
-# Installation
-You can install this package using pip:
-
-pip install ducklake-delta-exporter
-
-
-
-# Usage
-```
-from ducklake_delta_exporter import generate_latest_delta_log
-generate_latest_delta_log('path/to/your/ducklake.db', data_root='/lakehouse/default/Tables', checkpoint_interval=1)
-```

ducklake_delta_exporter-0.1.0/ducklake_delta_exporter.egg-info/PKG-INFO

@@ -1,52 +0,0 @@
-Metadata-Version: 2.4
-Name: ducklake-delta-exporter
-Version: 0.1.0
-Summary: A utility to export DuckLake database metadata to Delta Lake transaction logs.
-Home-page: https://github.com/djouallah/ducklake-delta-exporter
-Author: mim
-Author-email: your.email@example.com
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Intended Audience :: Developers
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Classifier: Development Status :: 3 - Alpha
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: duckdb
-Requires-Dist: pyarrow
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-
-# DuckLake Delta Exporter
-A Python utility to synchronize metadata from a DuckLake database with Delta Lake transaction logs. This allows you to manage data in DuckLake and make it discoverable and queryable by Delta Lake compatible tools (e.g., Spark, Delta Lake Rust/Python clients).
-
-# Features
-DuckLake to Delta Sync: Generates incremental Delta Lake transaction logs (_delta_log/*.json) and checkpoint files (_delta_log/*.checkpoint.parquet) based on the latest state of tables in a DuckLake database.
-
-Schema Mapping: Automatically maps DuckDB data types to their Spark SQL equivalents for Delta Lake schema definitions.
-
-Change Detection: Identifies added and removed data files since the last Delta export, ensuring only necessary updates are written to the log.
-
-Checkpointing: Supports creating Delta Lake checkpoint files at a configurable interval for efficient state reconstruction.
-
-# Installation
-You can install this package using pip:
-
-pip install ducklake-delta-exporter
-
-
-
-# Usage
-```
-from ducklake_delta_exporter import generate_latest_delta_log
-generate_latest_delta_log('path/to/your/ducklake.db', data_root='/lakehouse/default/Tables', checkpoint_interval=1)
-```