duckrun 0.0.0__tar.gz → 0.1.0__tar.gz

duckrun-0.1.0/PKG-INFO

@@ -0,0 +1,11 @@
+Metadata-Version: 2.4
+Name: duckrun
+Version: 0.1.0
+Summary: Lakehouse task runner powered by DuckDB for Microsoft Fabric
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/djouallah/duckrun
+Project-URL: Repository, https://github.com/djouallah/duckrun
+Project-URL: Issues, https://github.com/djouallah/duckrun/issues
+Requires-Python: >=3.9
+License-File: LICENSE
+Dynamic: license-file

duckrun-0.1.0/README.md

@@ -0,0 +1,162 @@
+# 🦆 Duckrun
+
+Simple lakehouse task runner for Microsoft Fabric, powered by DuckDB.
+
+## Installation
+
+```bash
+pip install duckrun
+```
+
+For local development (enables Azure CLI authentication):
+```bash
+pip install duckrun[local]
+```
+
+## Quick Start
+
+```python
+import duckrun as dr
+
+# Connect to your Fabric lakehouse
+lakehouse = dr.connect(
+    workspace="my_workspace",
+    lakehouse_name="my_lakehouse", 
+    schema="dbo",
+    sql_folder="./sql"  # folder containing your .sql and .py files
+)
+
+# Define your pipeline
+pipeline = [
+    ('load_data', (url, path)),           # Python task
+    ('clean_data', 'overwrite'),          # SQL task  
+    ('aggregate', 'append')               # SQL task
+]
+
+# Run it
+lakehouse.run(pipeline)
+```
+
+## How It Works
+
+Duckrun runs two types of tasks:
+
+### 1. Python Tasks
+Format: `('function_name', (arg1, arg2, ...))`
+
+Create a file `sql_folder/function_name.py` with a function matching the name:
+
+```python
+# sql_folder/load_data.py
+def load_data(url, path):
+    # your code here
+    return result
+```
+
+### 2. SQL Tasks  
+Format: `('table_name', 'mode')` or `('table_name', 'mode', {params})`
+
+Create a file `sql_folder/table_name.sql`:
+
+```sql
+-- sql_folder/clean_data.sql
+SELECT 
+    id,
+    TRIM(name) as name,
+    date
+FROM raw_data
+WHERE date >= '2024-01-01'
+```
+
+**Modes:**
+- `overwrite` - Replace table completely
+- `append` - Add to existing table
+- `ignore` - Create only if doesn't exist
+
+## SQL Parameters
+
+Your SQL files automatically have access to:
+- `$ws` - workspace name
+- `$lh` - lakehouse name
+- `$schema` - schema name
+
+Pass custom parameters:
+
+```python
+pipeline = [
+    ('sales', 'append', {'start_date': '2024-01-01', 'end_date': '2024-12-31'})
+]
+```
+
+```sql
+-- sql_folder/sales.sql
+SELECT * FROM transactions
+WHERE date BETWEEN '$start_date' AND '$end_date'
+```
+
+## Table Name Convention
+
+Use `__` to create variants of the same table:
+
+```python
+pipeline = [
+    ('sales__initial', 'overwrite', {}),    # writes to 'sales' table
+    ('sales__incremental', 'append', {}),   # appends to 'sales' table
+]
+```
+
+Both write to the same `sales` table, but use different SQL files.
+
+## Query Data
+
+```python
+# Run queries
+lakehouse.sql("SELECT * FROM my_table LIMIT 10").show()
+
+# Get as DataFrame
+df = lakehouse.sql("SELECT COUNT(*) FROM sales").df()
+```
+
+## Real-World Example
+
+```python
+import duckrun as dr
+
+lakehouse = dr.connect(
+    workspace="Analytics",
+    lakehouse_name="Sales", 
+    schema="dbo",
+    sql_folder="./etl"
+)
+
+# Daily pipeline
+daily = [
+    ('download_files', (api_url, local_path)),
+    ('staging_orders', 'overwrite', {'run_date': '2024-06-01'}),
+    ('staging_customers', 'overwrite', {'run_date': '2024-06-01'}),
+    ('fact_sales', 'append'),
+    ('dim_customer', 'overwrite')
+]
+
+lakehouse.run(daily)
+
+# Check results
+lakehouse.sql("SELECT COUNT(*) FROM fact_sales").show()
+```
+
+## Remote SQL Files
+
+You can load SQL/Python files from a URL:
+
+```python
+lakehouse = dr.connect(
+    workspace="Analytics",
+    lakehouse_name="Sales", 
+    schema="dbo",
+    sql_folder="https://raw.githubusercontent.com/user/repo/main/sql"
+)
+```
+
+## License
+
+MIT

duckrun-0.1.0/duckrun.egg-info/PKG-INFO

@@ -0,0 +1,11 @@
+Metadata-Version: 2.4
+Name: duckrun
+Version: 0.1.0
+Summary: Lakehouse task runner powered by DuckDB for Microsoft Fabric
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/djouallah/duckrun
+Project-URL: Repository, https://github.com/djouallah/duckrun
+Project-URL: Issues, https://github.com/djouallah/duckrun/issues
+Requires-Python: >=3.9
+License-File: LICENSE
+Dynamic: license-file

duckrun-0.1.0/pyproject.toml

@@ -0,0 +1,17 @@
+
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "duckrun"
+version = "0.1.0"
+description = "Lakehouse task runner powered by DuckDB for Microsoft Fabric"
+license = "MIT"
+requires-python = ">=3.9"
+
+[project.urls]
+Homepage = "https://github.com/djouallah/duckrun"
+Repository = "https://github.com/djouallah/duckrun"
+Issues = "https://github.com/djouallah/duckrun/issues"
+

duckrun-0.0.0/PKG-INFO

@@ -1,5 +0,0 @@
-Metadata-Version: 2.4
-Name: duckrun
-Version: 0.0.0
-License-File: LICENSE
-Dynamic: license-file

duckrun-0.0.0/README.md

@@ -1,39 +0,0 @@
-# 🦆 Duckrun
-
-Lakehouse task runner powered by DuckDB for Microsoft Fabric.
-
-## Features
-
-- 🦆 **DuckDB-powered**: Fast in-memory processing
-- 📦 **Delta Lake**: Native Delta table support
-- 🔄 **Simple API**: Clean tuple-based pipeline definition
-- 🎯 **Fabric-native**: Built for Microsoft Fabric lakehouses
-- 🐍 **Python + SQL**: Mix Python and SQL tasks seamlessly
-
-## Installation
-```bash
-pip install duckrun
-
-from duckrun import Duckrun
-
-# Connect to your lakehouse
-dr = Duckrun.connect(
-    workspace="your_workspace",
-    lakehouse_name="your_lakehouse",
-    schema="dbo",
-    sql_folder="./sql"
-)
-
-# Define pipeline
-pipeline = [
-    ('download', (urls, paths, depth)),
-    ('staging', 'overwrite', {'run_date': '2024-06-01'}),
-    ('transform', 'append'),
-    ('fact_sales', 'append')
-]
-
-# Run it
-dr.run(pipeline)
-
-# Query directly
-dr.sql("SELECT * FROM staging").show()

duckrun-0.0.0/duckrun.egg-info/PKG-INFO

@@ -1,5 +0,0 @@
-Metadata-Version: 2.4
-Name: duckrun
-Version: 0.0.0
-License-File: LICENSE
-Dynamic: license-file