sqlmesh-dag-generator 0.1.0__tar.gz

sqlmesh_dag_generator-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2025 SQLMesh DAG Generator Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

sqlmesh_dag_generator-0.1.0/PKG-INFO

@@ -0,0 +1,308 @@
+Metadata-Version: 2.4
+Name: sqlmesh-dag-generator
+Version: 0.1.0
+Summary: Open-source Airflow DAG generator for SQLMesh projects
+Home-page: https://github.com/kubolko/sqlmesh-dag-generator
+Author: Jakub Sumionka
+Author-email: jakub.sumionka@gmail.com
+Project-URL: Bug Reports, https://github.com/kubolko/sqlmesh-dag-generator/issues
+Project-URL: Source, https://github.com/kubolko/sqlmesh-dag-generator
+Project-URL: Documentation, https://github.com/kubolko/sqlmesh-dag-generator/docs
+Keywords: sqlmesh airflow dag generator etl data-engineering
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Code Generators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqlmesh>=0.20.0
+Requires-Dist: apache-airflow>=2.0.0
+Requires-Dist: pyyaml>=5.4.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SQLMesh DAG Generator
+
+Generate Apache Airflow DAGs from SQLMesh projects - **no cloud dependencies required**.
+
+Transform your SQLMesh models into production-ready Airflow DAGs with **full data lineage**, automatically!
+
+## ✨ Key Features
+
+- 🔥 **Dynamic DAG Generation (Default)**: Fire-and-forget - place DAG once, auto-discovers models at runtime
+- ✅ **Full Lineage in Airflow**: Each SQLMesh model = One Airflow task with proper dependencies
+- 🌍 **Multi-Environment Support**: Use Airflow Variables for dev/staging/prod
+- ⚡ **Incremental Models**: Proper handling with `data_interval_start/end`
+- 🎯 **Enhanced Error Handling**: SQLMesh-specific error messages in Airflow logs
+- 🛠️ **Dual Mode**: Dynamic (auto-discovery, default) or Static (full control)
+- 🚫 **No Vendor Lock-in**: Open source, no cloud dependencies
+
+## 🚀 Quick Start (3 Steps)
+
+### 1. Install
+```bash
+pip install sqlmesh-dag-generator  # (when published)
+# OR
+git clone <repo> && cd SQLMeshDAGGenerator && pip install -e .
+```
+
+### 2. Generate DAG (Dynamic Mode - Default!)
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+# Point to your SQLMesh project
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/your/sqlmesh/project",
+    dag_id="my_pipeline",
+    schedule_interval="@daily"
+)
+
+# Generate dynamic DAG (default - fire and forget!)
+dag_code = generator.generate_dynamic_dag()
+
+# Save it
+with open("my_pipeline.py", "w") as f:
+    f.write(dag_code)
+```
+
+### 3. Deploy to Airflow
+```bash
+cp my_pipeline.py /opt/airflow/dags/
+```
+
+**That's it! 🎉** Your SQLMesh models are now orchestrated by Airflow. The DAG will auto-discover models at runtime - no regeneration needed when models change!
+
+## 💡 What You Get
+
+### Your SQLMesh Project:
+```
+my_project/
+└── models/
+    ├── raw_orders.sql
+    ├── stg_orders.sql      # depends on raw_orders
+    └── orders_summary.sql  # depends on stg_orders
+```
+
+### Generated Airflow DAG:
+```
+Airflow Graph View:
+  [raw_orders] → [stg_orders] → [orders_summary]
+  
+✅ Each model = separate task
+✅ SQLMesh dependencies = Airflow dependencies  
+✅ Full lineage visible in Airflow UI
+```
+
+## 📚 Documentation
+
+- **[Quick Start Guide](docs/QUICKSTART.md)** - Step-by-step tutorial (start here!)
+- **[Usage Guide](docs/USAGE.md)** - Complete reference
+- **[Dynamic DAGs](docs/DYNAMIC_DAGS.md)** - Fire-and-forget mode explained
+- **[Examples](examples/)** - Code examples
+- **[Architecture](docs/ARCHITECTURE.md)** - Technical details
+
+## 🔥 Why Dynamic Mode (Default)?
+
+**Dynamic mode** auto-discovers SQLMesh models at runtime:
+
+```python
+dag_code = generator.generate_dynamic_dag()  # Default behavior!
+```
+
+**Benefits:**
+- ✅ **No regeneration needed** when SQLMesh models change
+- ✅ **Always in sync** - DAG updates automatically
+- ✅ **Multi-environment** - Uses Airflow Variables
+- ✅ **Production-ready** - Enhanced error handling
+
+Want static mode instead? Just use `generator.generate_dag()` - see [Usage Guide](docs/USAGE.md).
+
+## 🎯 Simple Example
+
+The simplest possible usage - just 3 lines of code:
+
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/your/sqlmesh/project",
+    dag_id="my_pipeline"
+)
+
+dag_code = generator.generate_dynamic_dag()
+```
+
+See [examples/simple_generate.py](examples/simple_generate.py) for a complete runnable example.
+
+## 🤝 Contributing
+
+Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+[Your License Here]
+
+---
+
+**Built with ❤️ for the data engineering community**
+
+### Configuration File
+
+Create a `dag_generator_config.yaml`:
+
+```yaml
+sqlmesh:
+  project_path: "/path/to/sqlmesh/project"
+  environment: "prod"
+  gateway: "local"
+
+airflow:
+  dag_id: "sqlmesh_pipeline"
+  schedule_interval: "0 0 * * *"
+  default_args:
+    owner: "data-team"
+    retries: 3
+    retry_delay_minutes: 5
+  tags:
+    - sqlmesh
+    - analytics
+
+generation:
+  output_dir: "/path/to/airflow/dags"
+  operator_type: "python"  # or "bash"
+  include_tests: true
+  parallel_tasks: true
+```
+
+## How It Works
+
+1. **Load SQLMesh Project**: Reads your SQLMesh project configuration and models
+2. **Extract Dependencies**: Analyzes SQL queries to build dependency graph
+3. **Generate Tasks**: Creates Airflow tasks for each SQLMesh model
+4. **Set Dependencies**: Connects tasks based on model dependencies
+5. **Apply Schedules**: Preserves cron schedules and execution logic
+6. **Output DAG**: Generates Python file ready for Airflow
+
+## Architecture
+
+```
+SQLMesh Project
+    ↓
+SQLMeshDAGGenerator
+    ├── Context Loader (loads SQLMesh context)
+    ├── Model Parser (extracts model metadata)
+    ├── Dependency Resolver (builds dependency graph)
+    └── DAG Builder (generates Airflow DAG)
+    ↓
+Airflow DAG File
+```
+
+## Advanced Features
+
+### Custom Operators
+
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+from airflow.operators.python import PythonOperator
+
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    custom_operator_class=PythonOperator,
+    operator_kwargs={"provide_context": True}
+)
+```
+
+### Model Filtering
+
+```python
+# Generate DAG for specific models only
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    include_models=["model1", "model2"],
+    exclude_models=["test_*"]
+)
+```
+
+### Dynamic Task Generation
+
+```python
+# Generate tasks with dynamic parallelism
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    enable_dynamic_tasks=True,
+    max_parallel_tasks=10
+)
+```
+
+## Requirements
+
+- Python >= 3.8
+- Apache Airflow >= 2.0
+- SQLMesh >= 0.20.0
+
+## Development
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/sqlmesh-dag-generator.git
+cd sqlmesh-dag-generator
+
+# Install in development mode
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run linter
+black .
+ruff check .
+```
+
+## Contributing
+
+Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Comparison with Tobiko Cloud
+
+| Feature | Tobiko Cloud | SQLMesh DAG Generator |
+|---------|-------------|----------------------|
+| Cost | Paid | **Free & Open Source** |
+| Deployment | Cloud-based | **Self-hosted** |
+| Customization | Limited | **Fully Customizable** |
+| Privacy | External | **On-premise** |
+| Dependencies | Cloud connection | **None** |
+
+## Support
+
+- 📖 [Documentation](https://github.com/yourusername/sqlmesh-dag-generator/docs)
+- 🐛 [Issue Tracker](https://github.com/yourusername/sqlmesh-dag-generator/issues)
+- 💬 [Discussions](https://github.com/yourusername/sqlmesh-dag-generator/discussions)
+

sqlmesh_dag_generator-0.1.0/README.md

@@ -0,0 +1,261 @@
+# SQLMesh DAG Generator
+
+Generate Apache Airflow DAGs from SQLMesh projects - **no cloud dependencies required**.
+
+Transform your SQLMesh models into production-ready Airflow DAGs with **full data lineage**, automatically!
+
+## ✨ Key Features
+
+- 🔥 **Dynamic DAG Generation (Default)**: Fire-and-forget - place DAG once, auto-discovers models at runtime
+- ✅ **Full Lineage in Airflow**: Each SQLMesh model = One Airflow task with proper dependencies
+- 🌍 **Multi-Environment Support**: Use Airflow Variables for dev/staging/prod
+- ⚡ **Incremental Models**: Proper handling with `data_interval_start/end`
+- 🎯 **Enhanced Error Handling**: SQLMesh-specific error messages in Airflow logs
+- 🛠️ **Dual Mode**: Dynamic (auto-discovery, default) or Static (full control)
+- 🚫 **No Vendor Lock-in**: Open source, no cloud dependencies
+
+## 🚀 Quick Start (3 Steps)
+
+### 1. Install
+```bash
+pip install sqlmesh-dag-generator  # (when published)
+# OR
+git clone <repo> && cd SQLMeshDAGGenerator && pip install -e .
+```
+
+### 2. Generate DAG (Dynamic Mode - Default!)
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+# Point to your SQLMesh project
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/your/sqlmesh/project",
+    dag_id="my_pipeline",
+    schedule_interval="@daily"
+)
+
+# Generate dynamic DAG (default - fire and forget!)
+dag_code = generator.generate_dynamic_dag()
+
+# Save it
+with open("my_pipeline.py", "w") as f:
+    f.write(dag_code)
+```
+
+### 3. Deploy to Airflow
+```bash
+cp my_pipeline.py /opt/airflow/dags/
+```
+
+**That's it! 🎉** Your SQLMesh models are now orchestrated by Airflow. The DAG will auto-discover models at runtime - no regeneration needed when models change!
+
+## 💡 What You Get
+
+### Your SQLMesh Project:
+```
+my_project/
+└── models/
+    ├── raw_orders.sql
+    ├── stg_orders.sql      # depends on raw_orders
+    └── orders_summary.sql  # depends on stg_orders
+```
+
+### Generated Airflow DAG:
+```
+Airflow Graph View:
+  [raw_orders] → [stg_orders] → [orders_summary]
+  
+✅ Each model = separate task
+✅ SQLMesh dependencies = Airflow dependencies  
+✅ Full lineage visible in Airflow UI
+```
+
+## 📚 Documentation
+
+- **[Quick Start Guide](docs/QUICKSTART.md)** - Step-by-step tutorial (start here!)
+- **[Usage Guide](docs/USAGE.md)** - Complete reference
+- **[Dynamic DAGs](docs/DYNAMIC_DAGS.md)** - Fire-and-forget mode explained
+- **[Examples](examples/)** - Code examples
+- **[Architecture](docs/ARCHITECTURE.md)** - Technical details
+
+## 🔥 Why Dynamic Mode (Default)?
+
+**Dynamic mode** auto-discovers SQLMesh models at runtime:
+
+```python
+dag_code = generator.generate_dynamic_dag()  # Default behavior!
+```
+
+**Benefits:**
+- ✅ **No regeneration needed** when SQLMesh models change
+- ✅ **Always in sync** - DAG updates automatically
+- ✅ **Multi-environment** - Uses Airflow Variables
+- ✅ **Production-ready** - Enhanced error handling
+
+Want static mode instead? Just use `generator.generate_dag()` - see [Usage Guide](docs/USAGE.md).
+
+## 🎯 Simple Example
+
+The simplest possible usage - just 3 lines of code:
+
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/your/sqlmesh/project",
+    dag_id="my_pipeline"
+)
+
+dag_code = generator.generate_dynamic_dag()
+```
+
+See [examples/simple_generate.py](examples/simple_generate.py) for a complete runnable example.
+
+## 🤝 Contributing
+
+Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+[Your License Here]
+
+---
+
+**Built with ❤️ for the data engineering community**
+
+### Configuration File
+
+Create a `dag_generator_config.yaml`:
+
+```yaml
+sqlmesh:
+  project_path: "/path/to/sqlmesh/project"
+  environment: "prod"
+  gateway: "local"
+
+airflow:
+  dag_id: "sqlmesh_pipeline"
+  schedule_interval: "0 0 * * *"
+  default_args:
+    owner: "data-team"
+    retries: 3
+    retry_delay_minutes: 5
+  tags:
+    - sqlmesh
+    - analytics
+
+generation:
+  output_dir: "/path/to/airflow/dags"
+  operator_type: "python"  # or "bash"
+  include_tests: true
+  parallel_tasks: true
+```
+
+## How It Works
+
+1. **Load SQLMesh Project**: Reads your SQLMesh project configuration and models
+2. **Extract Dependencies**: Analyzes SQL queries to build dependency graph
+3. **Generate Tasks**: Creates Airflow tasks for each SQLMesh model
+4. **Set Dependencies**: Connects tasks based on model dependencies
+5. **Apply Schedules**: Preserves cron schedules and execution logic
+6. **Output DAG**: Generates Python file ready for Airflow
+
+## Architecture
+
+```
+SQLMesh Project
+    ↓
+SQLMeshDAGGenerator
+    ├── Context Loader (loads SQLMesh context)
+    ├── Model Parser (extracts model metadata)
+    ├── Dependency Resolver (builds dependency graph)
+    └── DAG Builder (generates Airflow DAG)
+    ↓
+Airflow DAG File
+```
+
+## Advanced Features
+
+### Custom Operators
+
+```python
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+from airflow.operators.python import PythonOperator
+
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    custom_operator_class=PythonOperator,
+    operator_kwargs={"provide_context": True}
+)
+```
+
+### Model Filtering
+
+```python
+# Generate DAG for specific models only
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    include_models=["model1", "model2"],
+    exclude_models=["test_*"]
+)
+```
+
+### Dynamic Task Generation
+
+```python
+# Generate tasks with dynamic parallelism
+generator = SQLMeshDAGGenerator(
+    sqlmesh_project_path="/path/to/project",
+    enable_dynamic_tasks=True,
+    max_parallel_tasks=10
+)
+```
+
+## Requirements
+
+- Python >= 3.8
+- Apache Airflow >= 2.0
+- SQLMesh >= 0.20.0
+
+## Development
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/sqlmesh-dag-generator.git
+cd sqlmesh-dag-generator
+
+# Install in development mode
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run linter
+black .
+ruff check .
+```
+
+## Contributing
+
+Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Comparison with Tobiko Cloud
+
+| Feature | Tobiko Cloud | SQLMesh DAG Generator |
+|---------|-------------|----------------------|
+| Cost | Paid | **Free & Open Source** |
+| Deployment | Cloud-based | **Self-hosted** |
+| Customization | Limited | **Fully Customizable** |
+| Privacy | External | **On-premise** |
+| Dependencies | Cloud connection | **None** |
+
+## Support
+
+- 📖 [Documentation](https://github.com/yourusername/sqlmesh-dag-generator/docs)
+- 🐛 [Issue Tracker](https://github.com/yourusername/sqlmesh-dag-generator/issues)
+- 💬 [Discussions](https://github.com/yourusername/sqlmesh-dag-generator/discussions)
+

sqlmesh_dag_generator-0.1.0/pyproject.toml

@@ -0,0 +1,74 @@
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+addopts = [
+    "-v",
+    "--strict-markers",
+    "--tb=short",
+]
+
+[tool.coverage.run]
+source = ["sqlmesh_dag_generator"]
+omit = [
+    "*/tests/*",
+    "*/test_*.py",
+]
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "def __repr__",
+    "raise AssertionError",
+    "raise NotImplementedError",
+    "if __name__ == .__main__.:",
+    "if TYPE_CHECKING:",
+]
+
+[tool.black]
+line-length = 100
+target-version = ['py38', 'py39', 'py310', 'py311']
+include = '\.pyi?$'
+extend-exclude = '''
+/(
+  # directories
+  \.eggs
+  | \.git
+  | \.mypy_cache
+  | \.venv
+  | build
+  | dist
+)/
+'''
+
+[tool.ruff]
+line-length = 100
+target-version = "py38"
+select = [
+    "E",  # pycodestyle errors
+    "W",  # pycodestyle warnings
+    "F",  # pyflakes
+    "I",  # isort
+    "B",  # flake8-bugbear
+    "C4", # flake8-comprehensions
+    "UP", # pyupgrade
+]
+ignore = [
+    "E501",  # line too long, handled by black
+    "B008",  # do not perform function calls in argument defaults
+    "C901",  # too complex
+]
+
+[tool.mypy]
+python_version = "3.8"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = false
+disallow_incomplete_defs = false
+check_untyped_defs = true
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_no_return = true
+

sqlmesh_dag_generator-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+