pycharter 0.0.24__py3-none-any.whl → 0.0.26__py3-none-any.whl

pycharter/etl_generator/ASYNC_AND_EXECUTION.md

@@ -0,0 +1,91 @@
+# Async and Execution Model
+
+This document describes how PyCharter's ETL pipeline uses async execution, where the event loop runs, and how to run pipelines from scripts and long-running applications.
+
+## Pipeline execution is async
+
+The `Pipeline.run()` method is **async** and returns a coroutine. Extractors yield batches asynchronously, and loaders perform I/O asynchronously. You must run the pipeline within an event loop.
+
+## Running a pipeline
+
+### From a script (one-off run)
+
+Use `asyncio.run()` to run the pipeline. This creates an event loop, runs the pipeline, and closes the loop when done.
+
+```python
+import asyncio
+from pycharter import Pipeline
+
+async def main():
+    pipeline = Pipeline.from_config_dir("pipelines/users/")
+    result = await pipeline.run()
+    print(f"Loaded {result.rows_loaded} rows")
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+**Important:** Call `asyncio.run(main())` only once per process. Do not nest `asyncio.run()` calls.
+
+### From an async application (FastAPI, Celery async, etc.)
+
+If you are already inside an async context (e.g. a FastAPI route or an async Celery task), **await** the pipeline directly. Do not use `asyncio.run()` — it would create a new event loop and can conflict with the existing one.
+
+```python
+from fastapi import APIRouter
+from pycharter import Pipeline
+
+router = APIRouter()
+
+@router.post("/run-etl")
+async def run_etl():
+    pipeline = Pipeline.from_config_dir("pipelines/users/")
+    result = await pipeline.run()
+    return {"rows_loaded": result.rows_loaded}
+```
+
+### Where the event loop runs
+
+| Context                    | Event loop                    | How to run the pipeline      |
+|---------------------------|-------------------------------|------------------------------|
+| Script (e.g. `python run.py`) | Created by `asyncio.run()`     | `asyncio.run(main())`        |
+| Jupyter / IPython         | Built-in loop                 | `await pipeline.run()`       |
+| FastAPI / Starlette       | Uvicorn’s loop                | `await pipeline.run()`       |
+| Async Celery task         | Worker’s loop                 | `await pipeline.run()`       |
+| Sync code (no loop)       | None                          | Use `asyncio.run(main())`    |
+
+## Error handling and error mode
+
+`Pipeline.run()` accepts an optional `error_context` (from `pycharter.shared.errors`). The default error context’s mode controls whether failures **raise** or are **collected**:
+
+- **STRICT (default for many paths):** Extraction or load failures raise exceptions. Use when you want fail-fast behavior.
+- **LENIENT:** Failures are logged and appended to `result.errors`; the pipeline continues where possible.
+- **COLLECT:** Same as lenient but errors are also collected on the context for later inspection.
+
+```python
+from pycharter import Pipeline
+from pycharter.shared.errors import get_error_context, set_error_mode, ErrorMode
+
+# Optional: set global mode (e.g. lenient for a script)
+set_error_mode(ErrorMode.LENIENT)
+
+pipeline = Pipeline.from_config_dir("pipelines/users/")
+result = await pipeline.run()
+
+if not result.success:
+    for err in result.errors:
+        print(err)
+```
+
+You can also pass a specific `ErrorContext` into `run()` instead of using the global default:
+
+```python
+from pycharter.shared.errors import ErrorContext, ErrorMode
+
+ctx = ErrorContext(mode=ErrorMode.LENIENT)
+result = await pipeline.run(error_context=ctx)
+```
+
+## No async context manager (yet)
+
+The pipeline does not provide an `async with` context manager. Connections (e.g. DB, HTTP) are managed inside the extractor and loader. For cleanup, instantiate and run the pipeline in a scope where you control resource lifetime, or wrap the run in your own try/finally or async context manager.

pycharter/etl_generator/INTERFACES.md

@@ -0,0 +1,142 @@
+# ETL Generator — Main Interfaces
+
+This document describes the main public interfaces of `pycharter.etl_generator`. Use these when building or extending ETL pipelines.
+
+---
+
+## Primary interface: run pipelines
+
+### `ETLOrchestrator`
+
+**Import:** `from pycharter.etl_generator import ETLOrchestrator`
+
+The main entry point for running ETL. It runs: **Extract → Transform → Load** from contract artifacts and ETL configs (`extract.yaml`, `transform.yaml`, `load.yaml`).
+
+- **Constructor:** `ETLOrchestrator(contract_dir=None, contract_file=None, contract_dict=None, contract_metadata=None, checkpoint_dir=None, progress_callback=None, verbose=True, max_memory_mb=None, config_context=None, extract_config=None, transform_config=None, load_config=None, extract_file=None, transform_file=None, load_file=None)`
+- **Main methods:**
+  - `run(**kwargs)` → `Dict` — Run the full pipeline (async). Pass `dry_run=True` to transform/load without writing. Input params (e.g. `symbol`, `start_date`) come from `extract.yaml`’s `input_params` and `**kwargs`.
+  - `extract_stream(batch_size=None, max_records=None, **kwargs)` → `AsyncIterator[List[Dict]]` — Stream batches from extract only.
+- **Config sources (priority):** direct dict args > file paths > files in `contract_dir`.
+
+### `create_orchestrator(contract_dir=None, **kwargs) -> ETLOrchestrator`
+
+**Import:** `from pycharter.etl_generator import create_orchestrator`
+
+Factory helper that returns `ETLOrchestrator(contract_dir=contract_dir, **kwargs)`.
+
+---
+
+## Pipeline discovery
+
+### `PipelineFactory`
+
+**Import:** `from pycharter.etl_generator import PipelineFactory`
+
+Discovers pipelines from a root directory: each subdir that contains `extract.yaml`, `transform.yaml`, and `load.yaml` is treated as one pipeline.
+
+- **Constructor:** `PipelineFactory(config_root="configs", excluded_dirs=None, required_files=None)`
+- **Methods:**
+  - `get_pipeline_names() -> List[str]`
+  - `get_contract_dir(pipeline_name: str) -> Optional[str]`
+  - `create_orchestrator(pipeline_name, config_context=None, verbose=True, **orchestrator_kwargs) -> ETLOrchestrator`
+  - `refresh()` — Rescan `config_root`
+
+---
+
+## Extract
+
+### `extract_with_pagination_streaming(...) -> AsyncIterator[List[Dict]]`
+
+**Import:** `from pycharter.etl_generator.extractors import extract_with_pagination_streaming`
+
+Async generator that yields batches of records. Dispatches to the right extractor via `ExtractorFactory` using `type` (or auto-detection from `extract_config`).
+
+- **Args:** `extract_config`, `params`, `headers`, `contract_dir=None`, `batch_size=1000`, `max_records=None`, `config_context=None`
+
+### `ExtractorFactory.create(extract_config) -> BaseExtractor`
+
+**Import:** `from pycharter.etl_generator.extractors import ExtractorFactory`
+
+- `ExtractorFactory.create(extract_config)` — Returns an extractor instance for the given config.
+- **Auto-detection:** `base_url`/`api_endpoint` → http; `file_path` → file; `database` → database; `storage` → cloud_storage.
+
+### Extractors (implementations of `BaseExtractor`)
+
+**Import:** `from pycharter.etl_generator.extractors import HTTPExtractor, FileExtractor, DatabaseExtractor, CloudStorageExtractor, BaseExtractor`
+
+- **BaseExtractor** — Abstract base. Subclasses implement `validate_config(extract_config)` and `extract_streaming(extract_config, params, headers, contract_dir, batch_size, max_records, config_context)`.
+- **HTTPExtractor** — HTTP/API (single or paginated).
+- **FileExtractor** — Local files (CSV, JSON, Parquet, etc., including glob).
+- **DatabaseExtractor** — SQL over PostgreSQL, MySQL, SQLite, MSSQL, Oracle.
+- **CloudStorageExtractor** — S3, GCS, Azure Blob.
+
+Custom extractors: `ExtractorFactory.register(type_name, extractor_class)`.
+
+---
+
+## Transform
+
+### `apply_transforms(data, transform_config, **kwargs) -> List[Dict]`
+
+**Import:** `from pycharter.etl_generator.transformers import apply_transforms`
+
+Runs the transform pipeline on a list of records. Order: **simple_ops → jsonata → custom_function**. Each step is skipped if not present in config.
+
+- **Args:** `data: List[Dict]`, `transform_config: Dict`, plus any `**kwargs` passed to custom functions.
+- **Config shape:** Supports canonical `transform: { rename, convert, defaults, add, select, drop }` and/or top-level `jsonata`, `custom_function`. See `transformers.config.normalize_transform_config` and `pycharter/data/templates/etl/`.
+
+---
+
+## Load
+
+### `load_to_file(data, load_config, ...) -> Dict`
+
+**Import:** `from pycharter.etl_generator.loaders import load_to_file`
+
+Writes records to a local file. Used when `load_config` has `destination_type: file` (or implies it via `file_path`).
+
+- **Args:** `data`, `load_config`, `contract_dir=None`, `config_context=None`
+- **Returns:** `{ "written": n, "total": n }` (and related metadata).
+
+### `load_to_cloud_storage(data, load_config, ...) -> Dict`
+
+**Import:** `from pycharter.etl_generator.loaders import load_to_cloud_storage`
+
+Writes to S3, GCS, or Azure Blob. Used when `destination_type: cloud_storage` or config has `storage`.
+
+- **Args:** `data`, `load_config`, `contract_dir=None`, `config_context=None`
+- **Returns:** `{ "written": n, "total": n }` (and related metadata).
+
+Database loading is handled inside the orchestrator via `pycharter.etl_generator.database.load_data` when `destination_type` is omitted or `database` (and `load_config` has `database`, `target_table`, `schema_name`, etc.).
+
+---
+
+## Config generation
+
+### `generate_etl_config(...)` / `generate_etl_config_from_contract(...)` / `generate_etl_config_from_store(...)`
+
+**Import:** `from pycharter.etl_generator import generate_etl_config, generate_etl_config_from_contract, generate_etl_config_from_store`
+
+Helpers to produce ETL config dicts (extract/transform/load) from contracts or from the metadata store. See docstrings and `config_generator.py` for parameters.
+
+---
+
+## Utilities (progress, checkpoint, DLQ)
+
+- **Progress:** `ETLProgress`, `ProgressTracker` — `from pycharter.etl_generator import ETLProgress, ProgressTracker`
+- **Checkpoint/resume:** `CheckpointManager`, `CheckpointState` — `from pycharter.etl_generator import CheckpointManager, CheckpointState`
+- **Dead letter:** `DeadLetterQueue`, `DeadLetterRecord`, `DLQReason` — `from pycharter.etl_generator import DeadLetterQueue, DeadLetterRecord, DLQReason`
+
+---
+
+## Quick reference: import map
+
+| Use case                 | Import |
+|--------------------------|--------|
+| Run ETL                  | `from pycharter.etl_generator import ETLOrchestrator, create_orchestrator` |
+| Discover pipelines       | `from pycharter.etl_generator import PipelineFactory` |
+| Extract (streaming)      | `from pycharter.etl_generator.extractors import extract_with_pagination_streaming, ExtractorFactory` |
+| Transform                | `from pycharter.etl_generator.transformers import apply_transforms` |
+| Load to file/cloud       | `from pycharter.etl_generator.loaders import load_to_file, load_to_cloud_storage` |
+| Config generation        | `from pycharter.etl_generator import generate_etl_config, generate_etl_config_from_contract, generate_etl_config_from_store` |
+| Progress / checkpoint/DLQ| `from pycharter.etl_generator import ETLProgress, ProgressTracker, CheckpointManager, CheckpointState, DeadLetterQueue, DeadLetterRecord, DLQReason` |

pycharter/etl_generator/README.md

@@ -0,0 +1,271 @@
+# ETL Orchestrator - User Guide
+
+This document describes the ETL Orchestrator features, including simple transformations, JSONata support, and custom functions.
+
+## Transformation Capabilities
+
+The ETL orchestrator supports **three levels of transformation complexity**, applied in order:
+
+1. **Simple Operations** (declarative, easy to use) - NEW! ✅
+2. **JSONata** (powerful query language for complex transformations)
+3. **Custom Functions** (Python functions for advanced logic)
+
+### Simple Operations (Recommended for Most Use Cases)
+
+Simple, declarative operations that handle 90% of transformation needs:
+
+```yaml
+# transform.yaml
+transform:
+  rename:
+    oldName: new_name
+    camelCase: snake_case
+  convert:
+    price: float
+    quantity: integer
+  defaults:
+    status: "pending"
+  add:
+    full_name: "${first_name} ${last_name}"
+    created_at: "now()"
+  select:
+    - field1
+    - field2
+  drop:
+    - internal_id
+```
+
+**See [TRANSFORMATION_GUIDE.md](TRANSFORMATION_GUIDE.md) for complete documentation.**
+
+### JSONata (Advanced)
+
+Full JSONata support for complex transformations:
+
+```yaml
+jsonata:
+  expression: |
+    $.{
+      "ticker": symbol,
+      "avg_price": $average(prices),
+      "total_volume": $sum(volumes)
+    }
+  mode: "batch"  # or "record"
+```
+
+### Custom Functions
+
+Import and run external Python modules/functions:
+
+```yaml
+custom_function:
+  module: "myproject.transforms"
+  function: "optimize_data"
+  mode: "batch"
+  kwargs:
+    method: "min_volatility"
+```
+
+**All three can be used together!** Simple operations → JSONata → Custom functions.
+
+## Enhanced Features
+
+## Features Overview
+
+### Phase 1: Core Streaming Infrastructure ✅
+- **Streaming/Incremental ETL**: Process data in batches (Extract-Batch → Transform-Batch → Load-Batch)
+- **Generator-based Extraction**: Async generators for memory-efficient data extraction
+- **Memory Management**: Automatic memory monitoring and limits
+
+### Phase 2: Observability & Configuration ✅
+- **Configurable Processing Modes**: Choose between `full`, `streaming`, or `hybrid` modes
+- **Progress Tracking**: Real-time progress reporting with callbacks
+- **Error Recovery**: Retry strategies and error threshold management
+
+### Phase 3: Advanced Features ✅
+- **Checkpoint/Resume**: Save and resume long-running jobs
+- **Multiple Runs Support**: Process multiple parameter sets efficiently with rate limiting
+
+## Usage Examples
+
+### Basic Usage (Backward Compatible)
+
+```python
+from pycharter.etl_generator import ETLOrchestrator
+
+# Default behavior (full mode) - backward compatible
+orchestrator = ETLOrchestrator(contract_dir="data/examples/my_contract")
+result = await orchestrator.run()
+```
+
+### Streaming Mode (Memory Efficient)
+
+Configure in `extract.yaml`:
+```yaml
+processing_mode: streaming
+batch_size: 1000
+```
+
+Or use programmatically:
+```python
+orchestrator = ETLOrchestrator(contract_dir="data/examples/my_contract")
+result = await orchestrator.run_streaming(batch_size=1000)
+```
+
+### Progress Tracking
+
+```python
+from pycharter.etl_generator import ETLOrchestrator, ETLProgress
+
+def log_progress(progress: ETLProgress):
+    print(f"{progress}")
+
+orchestrator = ETLOrchestrator(
+    contract_dir="data/examples/my_contract",
+    progress_callback=log_progress,
+    verbose=True
+)
+result = await orchestrator.run()
+```
+
+### Checkpoint/Resume
+
+```python
+orchestrator = ETLOrchestrator(
+    contract_dir="data/examples/my_contract",
+    checkpoint_dir="./checkpoints"
+)
+
+# Run with checkpoint
+result = await orchestrator.run(checkpoint_id="my_job_001")
+
+# Resume from checkpoint
+result = await orchestrator.run(
+    checkpoint_id="my_job_001",
+    resume=True
+)
+```
+
+### Memory Management
+
+```python
+orchestrator = ETLOrchestrator(
+    contract_dir="data/examples/my_contract",
+    max_memory_mb=2048  # Limit to 2GB
+)
+result = await orchestrator.run()
+```
+
+### Multiple Runs Processing
+
+Run the same ETL pipeline multiple times with different parameters efficiently:
+
+```python
+orchestrator = ETLOrchestrator(contract_dir="data/examples/my_contract")
+
+# Simple case: vary a single parameter (e.g., symbols)
+results = await orchestrator.run_multiple(
+    param_name='symbol',
+    param_values=["AAPL", "MSFT", "GOOGL", "TSLA"],
+    batch_size=5,
+    delay_between_runs=1.0
+)
+
+for result in results:
+    params = result['params']
+    print(f"{params}: {result['success']} - {result.get('records', 0)} records")
+
+# Complex case: vary multiple parameters
+results = await orchestrator.run_multiple(
+    param_sets=[
+        {'symbol': 'AAPL', 'date': '2024-01-01'},
+        {'symbol': 'MSFT', 'date': '2024-01-02'},
+        {'symbol': 'GOOGL', 'date': '2024-01-03'},
+    ],
+    batch_size=3,
+    delay_between_runs=0.5
+)
+```
+
+### Error Recovery
+
+```python
+result = await orchestrator.run_streaming(
+    batch_size=1000,
+    max_retries=3,
+    error_threshold=0.1  # Abort if >10% of batches fail
+)
+```
+
+## Configuration Options
+
+### extract.yaml
+
+```yaml
+# Processing mode: 'full', 'streaming', or 'hybrid'
+processing_mode: streaming
+
+# Batch size for processing
+batch_size: 1000
+
+# Memory limit (optional)
+max_memory_mb: 2048
+
+# Checkpoint configuration (optional)
+checkpoint:
+  enabled: true
+  interval: 100  # Checkpoint every N batches
+```
+
+## Processing Modes
+
+### Full Mode (Default)
+- **Behavior**: Extract all → Transform all → Load all
+- **Use Case**: Small to medium datasets, backward compatible
+- **Memory**: All data in memory at once
+
+### Streaming Mode
+- **Behavior**: Extract-Batch → Transform-Batch → Load-Batch (incremental)
+- **Use Case**: Large datasets, memory-constrained environments
+- **Memory**: Constant memory usage (batch size)
+
+### Hybrid Mode
+- **Behavior**: Extract in chunks, transform/load in batches
+- **Use Case**: Fast extraction but slow transformation/loading
+- **Memory**: Moderate memory usage
+
+## Architecture
+
+### New Modules
+
+- **`progress.py`**: Progress tracking and observability
+- **`checkpoint.py`**: Checkpoint/resume functionality
+- **`extractors/`**: Modular extraction (HTTP, file, database, cloud) with streaming entry point `extract_with_pagination_streaming`
+- **`orchestrator.py`**: Enhanced with all new features
+
+### Backward Compatibility
+
+All new features are **100% backward compatible**. Existing code continues to work without changes. New features are opt-in via:
+- Configuration in `extract.yaml`
+- Optional constructor parameters
+- New method calls (`run_streaming()`, `run_multiple()`, etc.)
+
+## Performance Considerations
+
+### Memory Usage
+- **Full Mode**: O(n) where n = total records
+- **Streaming Mode**: O(b) where b = batch size
+- **Hybrid Mode**: O(c) where c = chunk size
+
+### Throughput
+- **Full Mode**: Fastest for small datasets (single pass)
+- **Streaming Mode**: Slower but handles unlimited size
+- **Hybrid Mode**: Balanced approach
+
+## Best Practices
+
+1. **Use streaming mode** for datasets > 100K records
+2. **Enable checkpoints** for jobs expected to run > 1 hour
+3. **Set memory limits** to prevent OOM crashes
+4. **Use progress callbacks** for monitoring long-running jobs
+5. **Configure error thresholds** based on data quality expectations
+