sqlmesh-dag-generator 0.9.0__tar.gz → 0.9.6__tar.gz

{sqlmesh_dag_generator-0.9.0/sqlmesh_dag_generator.egg-info → sqlmesh_dag_generator-0.9.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlmesh-dag-generator
-Version: 0.9.0
+Version: 0.9.6
 Summary: Open-source Airflow DAG generator for SQLMesh projects
 Home-page: https://github.com/kubolko/sqlmesh-dag-generator
 Author: Jakub Sumionka
@@ -59,38 +59,30 @@ Transform your SQLMesh models into production-ready Airflow DAGs with **full dat
 - ✅ **Full Lineage in Airflow**: Each SQLMesh model = One Airflow task with proper dependencies
 - 🌍 **Multi-Environment Support**: Use Airflow Variables + SQLMesh gateways for dev/staging/prod
 - ⚡ **Incremental Models**: Proper handling with `data_interval_start/end`
+- 🩺 **Integrity Guardrails**: Warn when sub-hourly incremental models run with `catchup=False`, with optional bounded recovery helpers
 - 🎯 **Enhanced Error Handling**: SQLMesh-specific error messages in Airflow logs
 - 🛠️ **Dual Mode**: Dynamic (auto-discovery, default) or Static (full control)
 - 🚫 **No Vendor Lock-in**: Open source, no cloud dependencies
 
-### 🏢 Enterprise Features (NEW in v0.8.0)
-
-- 🔔 **Callbacks**: `on_failure_callback`, `on_success_callback`, `sla_miss_callback` for alerting
-- 🏷️ **Tag-Based Filtering**: `include_tags`, `exclude_tags` for Data Mesh team-specific DAGs
-- 🎱 **Pool Configuration**: `pool`, `pool_slots` for resource management
-- ⏩ **Trigger Downstream DAGs**: `trigger_dag_id` for ML pipeline integration
-- 🎯 **Pattern Filtering**: `model_pattern` for regex-based model selection
-
 ## ⚠️ Important: Gateway vs Environment
 
 **SQLMesh uses "gateways" to switch between environments, NOT an "environment" parameter.**
 
 ```python
-# ❌ WRONG - environment parameter is for SQLMesh virtual environments (testing)
+# ❌ WRONG - environment parameter is deprecated
 generator = SQLMeshDAGGenerator(
     sqlmesh_project_path="/path/to/project",
-    environment="prod",  # ERROR: Environment 'prod' was not found
+    environment="prod",  # This doesn't work!
 )
 
-# ✅ CORRECT - Use gateway to switch between dev/staging/prod
+# ✅ CORRECT - Use gateway to switch environments
 generator = SQLMeshDAGGenerator(
     sqlmesh_project_path="/path/to/project",
-    gateway="prod",  # This is how you select your environment!
-    # environment defaults to "" (empty string) - perfect for production
+    gateway="prod"  # This is how you select your environment!
 )
 ```
 
-**📚 Read [Understanding SQLMesh Environments](docs/SQLMESH_ENVIRONMENTS.md) for the complete explanation of the difference between `gateway` and `environment`.**
+**See [Multi-Environment Configuration Guide](docs/MULTI_ENVIRONMENT.md) for complete setup instructions.**
 
 ## 🚀 Quick Start (3 Steps)
 
@@ -127,6 +119,72 @@ cp my_pipeline.py /opt/airflow/dags/
 
 **That's it! 🎉** Your SQLMesh models are now orchestrated by Airflow. The DAG will auto-discover models at runtime - no regeneration needed when models change!
 
+## Recovery And Completeness
+
+SQLMesh DAG Generator forwards Airflow's `data_interval_start` and `data_interval_end` into `ctx.run(start=..., end=...)`.
+That means the package executes the interval Airflow gives it, but it does **not** invent missed Airflow runs on its own.
+
+If you run sub-hourly incremental models with `catchup=False`, outages can leave completeness gaps unless you replay the missed windows.
+
+The package now supports an explicit recovery policy:
+
+- `recovery_mode="disabled"`: no runtime recovery tasks are added.
+- `recovery_mode="warn"`: add an integrity guard task that detects missing intervals and logs them.
+- `recovery_mode="bounded_auto"` (default): add the same guard task plus a bounded recovery task that replays missing intervals when the gap is within `recovery_max_intervals`.
+
+Example:
+
+```python
+generator = SQLMeshDAGGenerator(
+  sqlmesh_project_path="/path/to/project",
+  dag_id="my_pipeline",
+  recovery_mode="bounded_auto",
+  recovery_max_intervals=6,
+)
+```
+
+When `recovery_mode` is enabled, the package adds stable helper tasks to the DAG instead of mutating the graph at runtime:
+
+- `sqlmesh_integrity_guard`
+- `sqlmesh_recovery_backfill` in `bounded_auto` mode
+
+This keeps recovery explicit and observable in Airflow while preserving the default "no surprise backfills" behavior.
+
+### Manual Backfill Task
+
+For larger historical gaps, the package also exposes a first-class manual backfill task helper.
+Use it in a separate unscheduled DAG and trigger it only when you need to replay a broader window:
+
+```python
+from datetime import datetime, timedelta
+from airflow import DAG
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+generator = SQLMeshDAGGenerator(
+  sqlmesh_project_path="/path/to/project",
+  dag_id="my_pipeline",
+  gateway="prod",
+)
+
+with DAG(
+  dag_id="my_pipeline_manual_backfill",
+  schedule=None,
+  start_date=datetime(2024, 1, 1),
+  catchup=False,
+) as dag:
+  generator.create_manual_backfill_task(
+    dag,
+    default_start="2024-01-01T00:00:00",
+    execution_timeout=timedelta(hours=12),
+  )
+```
+
+The task reads optional `dag_run.conf` overrides:
+
+- `start`: ISO-8601 start boundary
+- `end`: ISO-8601 end boundary. If omitted, the task backfills up to the current UTC time.
+- `models`: one model name or a list of model names. If omitted, the task backfills all models.
+
 ## 💡 What You Get
 
 ### Your SQLMesh Project:
@@ -150,27 +208,17 @@ Airflow Graph View:
 
 ## 📚 Documentation
 
-### Getting Started
 - **[Quick Start Guide](docs/QUICKSTART.md)** - Step-by-step tutorial (start here!)
 - **[Quick Reference](docs/QUICK_REFERENCE.md)** - One-page cheat sheet
-- **[Examples](examples/)** - Code examples
-
-### Configuration & Features
-- **[Auto-Scheduling Guide](docs/AUTO_SCHEDULING.md)** - Automatic schedule detection 📅
+- **[Auto-Scheduling Guide](docs/AUTO_SCHEDULING.md)** - Automatic schedule detection 📅 NEW!
 - **[Runtime Configuration](docs/RUNTIME_CONFIGURATION.md)** - Pass credentials via Airflow Connections 🔐
 - **[Multi-Environment Setup](docs/MULTI_ENVIRONMENT.md)** - Configure for dev/staging/prod ⚠️ IMPORTANT
+- **[Migration Guide](docs/MIGRATION_GUIDE.md)** - Fix common configuration issues
+- **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Common issues and solutions
+- **[Usage Guide](docs/USAGE.md)** - Complete reference
 - **[Dynamic DAGs](docs/DYNAMIC_DAGS.md)** - Fire-and-forget mode explained
-
-### Deployment & Production
-- **[Deployment Checklist](docs/DEPLOYMENT_CHECKLIST.md)** - Step-by-step deployment guide ✅ NEW!
-- **[Container Deployment](docs/CONTAINER_DEPLOYMENT.md)** - Docker, Kubernetes, ECS, Cloud Composer 🐳 NEW!
-- **[Common Issues](docs/COMMON_ISSUES.md)** - Real-world problems and solutions 🔧 NEW!
-- **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Debug guide
 - **[Deployment Warnings](docs/DEPLOYMENT_WARNINGS.md)** - Critical production considerations
-
-### Reference
-- **[Usage Guide](docs/USAGE.md)** - Complete reference
-- **[Migration Guide](docs/MIGRATION_GUIDE.md)** - Fix common configuration issues
+- **[Examples](examples/)** - Code examples
 - **[Architecture](docs/ARCHITECTURE.md)** - Technical details
 
 ## 🔥 Why Dynamic Mode (Default)?
@@ -225,9 +273,8 @@ Create a `dag_generator_config.yaml`:
 ```yaml
 sqlmesh:
   project_path: "/path/to/sqlmesh/project"
-  gateway: "prod"
-  environment: ""  # Empty for production
-  default_catalog: "my_catalog" # Optional: For 3-part naming support
+  environment: "prod"
+  gateway: "local"
 
 airflow:
   dag_id: "sqlmesh_pipeline"
@@ -245,12 +292,6 @@ generation:
   operator_type: "python"  # or "bash"
   include_tests: true
   parallel_tasks: true
-  auto_replan_on_change: true  # Automatically run 'sqlmesh plan' if models change
-  # Plan optimization options (for faster execution when no changes)
-  skip_backfill: false   # Skip apply if backfill is required (use CI/CD for initial backfills)
-  plan_only: false       # Generate plan without applying (for review/dry-run)
-  log_plan_details: true # Log detailed plan information (context, plan, apply phases)
-  return_value: true     # Enable/disable XCom return values
 ```
 
 ## How It Works
@@ -370,115 +411,6 @@ black .
 ruff check .
 ```
 
-## 🔧 Troubleshooting
-
-### Common Issues
-
-#### 1. `TypeError: Object of type CompletionStatus is not JSON serializable`
-**Cause:** Airflow cannot serialize the SQLMesh `CompletionStatus` enum returned by tasks.
-**Fix:** Upgrade to v0.7.2+. This version automatically converts the status to a string.
-Alternatively, use `return_value=False` in your generator config:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    return_value=False  # Disable XCom return values
-)
-```
-
-#### 2. `ConfigError: A query is required and must be a SELECT statement` (External Models)
-**Cause:** Defining external models in `.sql` files is not supported by the generator.
-**Fix:** Define external models in `external_models.yaml` instead.
-```yaml
-# external_models.yaml
-models:
-  - name: raw.users
-    kind: EXTERNAL
-    columns:
-      id: INT
-      name: TEXT
-```
-
-#### 3. "Environment 'prod' was not found"
-**Cause:** You are using `environment="prod"` instead of `gateway="prod"`.
-**Fix:** Set `gateway="prod"` and leave `environment` as empty string (default).
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    gateway="prod",
-    environment=""  # Correct for production
-)
-```
-
-#### 4. Changes to models are not picked up
-**Cause:** SQLMesh requires a plan/apply step to register changes.
-**Fix:** Enable `auto_replan_on_change` in your generator config:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    auto_replan_on_change=True  # Automatically runs 'sqlmesh plan --auto-apply'
-)
-```
-Or run `sqlmesh plan --auto-apply` manually.
-
-#### 5. `sqlmesh_plan_apply` task takes 7+ minutes even with no changes
-**Cause:** When `auto_replan_on_change=True`, the plan task may run full backfills even when there are no model changes.
-**Fix (v0.9.0+):** The plan optimization now:
-- ✅ Detects "no changes" scenarios and skips apply (~15 seconds instead of 7+ minutes)
-- ✅ Logs detailed timing for each phase (context load, plan compute, apply)
-
-For additional control, use these options:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    auto_replan_on_change=True,
-    skip_backfill=True,    # Skip apply if backfill is required (use CI/CD for initial backfills)
-    plan_only=True,        # Generate plan without applying (for review/dry-run)
-    log_plan_details=True  # Log detailed plan information (default: True)
-)
-```
-
-**Recommended CI/CD approach:**
-- Run `sqlmesh plan --auto-apply` in CI/CD on model changes (handles backfills)
-- Set `skip_backfill=True` in Airflow DAG (only applies non-backfill changes)
-- This gives you fast DAG runs (~15s) while ensuring changes are still applied
-
-#### 5. 3-Part Table Naming Issues (Cross-Database)
-**Cause:** SQLMesh generates queries with 3-part names (catalog.schema.table) which might fail if the catalog name doesn't match.
-**Fix:** Set `default_catalog` in your config to tell SQLMesh which catalog to omit or use.
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    default_catalog="my_catalog"
-)
-```
-
-### Advanced Features
-
-#### Health Check Task
-Add a pre-flight check to verify database connectivity before running models:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    enable_health_check=True
-)
-```
-
-#### Skip Audits
-For faster development iterations, you can skip audit checks:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    skip_audits=True
-)
-```
-
-#### Partial DAG Runs
-You can generate tasks for a subset of models (useful for testing):
-```python
-# In your DAG file
-generator.create_tasks_in_dag(dag, models=["model_a", "model_b"])
-```
-
 ## Contributing
 
 Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.

{sqlmesh_dag_generator-0.9.0 → sqlmesh_dag_generator-0.9.6}/README.md

@@ -12,38 +12,30 @@ Transform your SQLMesh models into production-ready Airflow DAGs with **full dat
 - ✅ **Full Lineage in Airflow**: Each SQLMesh model = One Airflow task with proper dependencies
 - 🌍 **Multi-Environment Support**: Use Airflow Variables + SQLMesh gateways for dev/staging/prod
 - ⚡ **Incremental Models**: Proper handling with `data_interval_start/end`
+- 🩺 **Integrity Guardrails**: Warn when sub-hourly incremental models run with `catchup=False`, with optional bounded recovery helpers
 - 🎯 **Enhanced Error Handling**: SQLMesh-specific error messages in Airflow logs
 - 🛠️ **Dual Mode**: Dynamic (auto-discovery, default) or Static (full control)
 - 🚫 **No Vendor Lock-in**: Open source, no cloud dependencies
 
-### 🏢 Enterprise Features (NEW in v0.8.0)
-
-- 🔔 **Callbacks**: `on_failure_callback`, `on_success_callback`, `sla_miss_callback` for alerting
-- 🏷️ **Tag-Based Filtering**: `include_tags`, `exclude_tags` for Data Mesh team-specific DAGs
-- 🎱 **Pool Configuration**: `pool`, `pool_slots` for resource management
-- ⏩ **Trigger Downstream DAGs**: `trigger_dag_id` for ML pipeline integration
-- 🎯 **Pattern Filtering**: `model_pattern` for regex-based model selection
-
 ## ⚠️ Important: Gateway vs Environment
 
 **SQLMesh uses "gateways" to switch between environments, NOT an "environment" parameter.**
 
 ```python
-# ❌ WRONG - environment parameter is for SQLMesh virtual environments (testing)
+# ❌ WRONG - environment parameter is deprecated
 generator = SQLMeshDAGGenerator(
     sqlmesh_project_path="/path/to/project",
-    environment="prod",  # ERROR: Environment 'prod' was not found
+    environment="prod",  # This doesn't work!
 )
 
-# ✅ CORRECT - Use gateway to switch between dev/staging/prod
+# ✅ CORRECT - Use gateway to switch environments
 generator = SQLMeshDAGGenerator(
     sqlmesh_project_path="/path/to/project",
-    gateway="prod",  # This is how you select your environment!
-    # environment defaults to "" (empty string) - perfect for production
+    gateway="prod"  # This is how you select your environment!
 )
 ```
 
-**📚 Read [Understanding SQLMesh Environments](docs/SQLMESH_ENVIRONMENTS.md) for the complete explanation of the difference between `gateway` and `environment`.**
+**See [Multi-Environment Configuration Guide](docs/MULTI_ENVIRONMENT.md) for complete setup instructions.**
 
 ## 🚀 Quick Start (3 Steps)
 
@@ -80,6 +72,72 @@ cp my_pipeline.py /opt/airflow/dags/
 
 **That's it! 🎉** Your SQLMesh models are now orchestrated by Airflow. The DAG will auto-discover models at runtime - no regeneration needed when models change!
 
+## Recovery And Completeness
+
+SQLMesh DAG Generator forwards Airflow's `data_interval_start` and `data_interval_end` into `ctx.run(start=..., end=...)`.
+That means the package executes the interval Airflow gives it, but it does **not** invent missed Airflow runs on its own.
+
+If you run sub-hourly incremental models with `catchup=False`, outages can leave completeness gaps unless you replay the missed windows.
+
+The package now supports an explicit recovery policy:
+
+- `recovery_mode="disabled"`: no runtime recovery tasks are added.
+- `recovery_mode="warn"`: add an integrity guard task that detects missing intervals and logs them.
+- `recovery_mode="bounded_auto"` (default): add the same guard task plus a bounded recovery task that replays missing intervals when the gap is within `recovery_max_intervals`.
+
+Example:
+
+```python
+generator = SQLMeshDAGGenerator(
+  sqlmesh_project_path="/path/to/project",
+  dag_id="my_pipeline",
+  recovery_mode="bounded_auto",
+  recovery_max_intervals=6,
+)
+```
+
+When `recovery_mode` is enabled, the package adds stable helper tasks to the DAG instead of mutating the graph at runtime:
+
+- `sqlmesh_integrity_guard`
+- `sqlmesh_recovery_backfill` in `bounded_auto` mode
+
+This keeps recovery explicit and observable in Airflow while preserving the default "no surprise backfills" behavior.
+
+### Manual Backfill Task
+
+For larger historical gaps, the package also exposes a first-class manual backfill task helper.
+Use it in a separate unscheduled DAG and trigger it only when you need to replay a broader window:
+
+```python
+from datetime import datetime, timedelta
+from airflow import DAG
+from sqlmesh_dag_generator import SQLMeshDAGGenerator
+
+generator = SQLMeshDAGGenerator(
+  sqlmesh_project_path="/path/to/project",
+  dag_id="my_pipeline",
+  gateway="prod",
+)
+
+with DAG(
+  dag_id="my_pipeline_manual_backfill",
+  schedule=None,
+  start_date=datetime(2024, 1, 1),
+  catchup=False,
+) as dag:
+  generator.create_manual_backfill_task(
+    dag,
+    default_start="2024-01-01T00:00:00",
+    execution_timeout=timedelta(hours=12),
+  )
+```
+
+The task reads optional `dag_run.conf` overrides:
+
+- `start`: ISO-8601 start boundary
+- `end`: ISO-8601 end boundary. If omitted, the task backfills up to the current UTC time.
+- `models`: one model name or a list of model names. If omitted, the task backfills all models.
+
 ## 💡 What You Get
 
 ### Your SQLMesh Project:
@@ -103,27 +161,17 @@ Airflow Graph View:
 
 ## 📚 Documentation
 
-### Getting Started
 - **[Quick Start Guide](docs/QUICKSTART.md)** - Step-by-step tutorial (start here!)
 - **[Quick Reference](docs/QUICK_REFERENCE.md)** - One-page cheat sheet
-- **[Examples](examples/)** - Code examples
-
-### Configuration & Features
-- **[Auto-Scheduling Guide](docs/AUTO_SCHEDULING.md)** - Automatic schedule detection 📅
+- **[Auto-Scheduling Guide](docs/AUTO_SCHEDULING.md)** - Automatic schedule detection 📅 NEW!
 - **[Runtime Configuration](docs/RUNTIME_CONFIGURATION.md)** - Pass credentials via Airflow Connections 🔐
 - **[Multi-Environment Setup](docs/MULTI_ENVIRONMENT.md)** - Configure for dev/staging/prod ⚠️ IMPORTANT
+- **[Migration Guide](docs/MIGRATION_GUIDE.md)** - Fix common configuration issues
+- **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Common issues and solutions
+- **[Usage Guide](docs/USAGE.md)** - Complete reference
 - **[Dynamic DAGs](docs/DYNAMIC_DAGS.md)** - Fire-and-forget mode explained
-
-### Deployment & Production
-- **[Deployment Checklist](docs/DEPLOYMENT_CHECKLIST.md)** - Step-by-step deployment guide ✅ NEW!
-- **[Container Deployment](docs/CONTAINER_DEPLOYMENT.md)** - Docker, Kubernetes, ECS, Cloud Composer 🐳 NEW!
-- **[Common Issues](docs/COMMON_ISSUES.md)** - Real-world problems and solutions 🔧 NEW!
-- **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Debug guide
 - **[Deployment Warnings](docs/DEPLOYMENT_WARNINGS.md)** - Critical production considerations
-
-### Reference
-- **[Usage Guide](docs/USAGE.md)** - Complete reference
-- **[Migration Guide](docs/MIGRATION_GUIDE.md)** - Fix common configuration issues
+- **[Examples](examples/)** - Code examples
 - **[Architecture](docs/ARCHITECTURE.md)** - Technical details
 
 ## 🔥 Why Dynamic Mode (Default)?
@@ -178,9 +226,8 @@ Create a `dag_generator_config.yaml`:
 ```yaml
 sqlmesh:
   project_path: "/path/to/sqlmesh/project"
-  gateway: "prod"
-  environment: ""  # Empty for production
-  default_catalog: "my_catalog" # Optional: For 3-part naming support
+  environment: "prod"
+  gateway: "local"
 
 airflow:
   dag_id: "sqlmesh_pipeline"
@@ -198,12 +245,6 @@ generation:
   operator_type: "python"  # or "bash"
   include_tests: true
   parallel_tasks: true
-  auto_replan_on_change: true  # Automatically run 'sqlmesh plan' if models change
-  # Plan optimization options (for faster execution when no changes)
-  skip_backfill: false   # Skip apply if backfill is required (use CI/CD for initial backfills)
-  plan_only: false       # Generate plan without applying (for review/dry-run)
-  log_plan_details: true # Log detailed plan information (context, plan, apply phases)
-  return_value: true     # Enable/disable XCom return values
 ```
 
 ## How It Works
@@ -323,115 +364,6 @@ black .
 ruff check .
 ```
 
-## 🔧 Troubleshooting
-
-### Common Issues
-
-#### 1. `TypeError: Object of type CompletionStatus is not JSON serializable`
-**Cause:** Airflow cannot serialize the SQLMesh `CompletionStatus` enum returned by tasks.
-**Fix:** Upgrade to v0.7.2+. This version automatically converts the status to a string.
-Alternatively, use `return_value=False` in your generator config:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    return_value=False  # Disable XCom return values
-)
-```
-
-#### 2. `ConfigError: A query is required and must be a SELECT statement` (External Models)
-**Cause:** Defining external models in `.sql` files is not supported by the generator.
-**Fix:** Define external models in `external_models.yaml` instead.
-```yaml
-# external_models.yaml
-models:
-  - name: raw.users
-    kind: EXTERNAL
-    columns:
-      id: INT
-      name: TEXT
-```
-
-#### 3. "Environment 'prod' was not found"
-**Cause:** You are using `environment="prod"` instead of `gateway="prod"`.
-**Fix:** Set `gateway="prod"` and leave `environment` as empty string (default).
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    gateway="prod",
-    environment=""  # Correct for production
-)
-```
-
-#### 4. Changes to models are not picked up
-**Cause:** SQLMesh requires a plan/apply step to register changes.
-**Fix:** Enable `auto_replan_on_change` in your generator config:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    auto_replan_on_change=True  # Automatically runs 'sqlmesh plan --auto-apply'
-)
-```
-Or run `sqlmesh plan --auto-apply` manually.
-
-#### 5. `sqlmesh_plan_apply` task takes 7+ minutes even with no changes
-**Cause:** When `auto_replan_on_change=True`, the plan task may run full backfills even when there are no model changes.
-**Fix (v0.9.0+):** The plan optimization now:
-- ✅ Detects "no changes" scenarios and skips apply (~15 seconds instead of 7+ minutes)
-- ✅ Logs detailed timing for each phase (context load, plan compute, apply)
-
-For additional control, use these options:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    auto_replan_on_change=True,
-    skip_backfill=True,    # Skip apply if backfill is required (use CI/CD for initial backfills)
-    plan_only=True,        # Generate plan without applying (for review/dry-run)
-    log_plan_details=True  # Log detailed plan information (default: True)
-)
-```
-
-**Recommended CI/CD approach:**
-- Run `sqlmesh plan --auto-apply` in CI/CD on model changes (handles backfills)
-- Set `skip_backfill=True` in Airflow DAG (only applies non-backfill changes)
-- This gives you fast DAG runs (~15s) while ensuring changes are still applied
-
-#### 5. 3-Part Table Naming Issues (Cross-Database)
-**Cause:** SQLMesh generates queries with 3-part names (catalog.schema.table) which might fail if the catalog name doesn't match.
-**Fix:** Set `default_catalog` in your config to tell SQLMesh which catalog to omit or use.
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    default_catalog="my_catalog"
-)
-```
-
-### Advanced Features
-
-#### Health Check Task
-Add a pre-flight check to verify database connectivity before running models:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    enable_health_check=True
-)
-```
-
-#### Skip Audits
-For faster development iterations, you can skip audit checks:
-```python
-generator = SQLMeshDAGGenerator(
-    ...,
-    skip_audits=True
-)
-```
-
-#### Partial DAG Runs
-You can generate tasks for a subset of models (useful for testing):
-```python
-# In your DAG file
-generator.create_tasks_in_dag(dag, models=["model_a", "model_b"])
-```
-
 ## Contributing
 
 Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.

{sqlmesh_dag_generator-0.9.0 → sqlmesh_dag_generator-0.9.6}/setup.py

@@ -20,7 +20,7 @@ if requirements_file.exists():
 
 setup(
     name="sqlmesh-dag-generator",
-    version="0.9.0",
+    version="0.9.6",
     description="Open-source Airflow DAG generator for SQLMesh projects",
     long_description=long_description,
     long_description_content_type="text/markdown",

{sqlmesh_dag_generator-0.9.0 → sqlmesh_dag_generator-0.9.6}/sqlmesh_dag_generator/__init__.py

@@ -2,11 +2,11 @@
 SQLMesh DAG Generator - Open Source Airflow Integration for SQLMesh
 """
 
-__version__ = "0.8.3"
+__version__ = "0.9.6"
 
 
 from sqlmesh_dag_generator.generator import SQLMeshDAGGenerator
-from sqlmesh_dag_generator.config import DAGGeneratorConfig
+from sqlmesh_dag_generator.config import DAGGeneratorConfig, RecoveryConfig
 from sqlmesh_dag_generator.airflow_utils import (
     resolve_credentials,
     register_credential_resolver,
@@ -16,6 +16,7 @@ from sqlmesh_dag_generator.airflow_utils import (
 __all__ = [
     "SQLMeshDAGGenerator",
     "DAGGeneratorConfig",
+    "RecoveryConfig",
     "resolve_credentials",
     "register_credential_resolver",
     "CredentialResolver",