pycharter 0.0.22__py3-none-any.whl → 0.0.24__py3-none-any.whl

{pycharter-0.0.22.dist-info → pycharter-0.0.24.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pycharter
-Version: 0.0.22
+Version: 0.0.24
 Summary: A Python package for data contract management with five core services: contract parsing, metadata storage, Pydantic generation, JSON Schema conversion, and runtime validation
 Author-email: semantic developers <na@example.com>
 License-Expression: MIT
@@ -54,6 +54,11 @@ Requires-Dist: pyspark>=3.5.0; extra == "worker"
 Requires-Dist: redis>=5.0.0; extra == "worker"
 Provides-Extra: etl
 Requires-Dist: sshtunnel>=0.4.0; extra == "etl"
+Requires-Dist: boto3>=1.26.0; extra == "etl"
+Requires-Dist: google-cloud-storage>=2.0.0; extra == "etl"
+Requires-Dist: azure-storage-blob>=12.0.0; extra == "etl"
+Requires-Dist: openpyxl>=3.0.0; extra == "etl"
+Requires-Dist: lxml>=4.9.0; extra == "etl"
 Dynamic: license-file
 
 # PyCharter
@@ -159,6 +164,41 @@ pycharter ui dev     # Development mode with hot reload
 
 ## 🚀 Quick Start
 
+### Quick Start: ETL Pipelines
+
+Build and run ETL pipelines programmatically (with the `|` operator) or from YAML configs. Pipeline **run()** is async; use `asyncio.run()` from scripts or `await` in async code.
+
+```python
+import asyncio
+from pycharter import Pipeline, HTTPExtractor, PostgresLoader, Rename, AddField
+
+# Programmatic pipeline
+pipeline = (
+    Pipeline(HTTPExtractor(url="https://api.example.com/data"))
+    | Rename({"old": "new"})
+    | AddField("processed_at", "now()")
+    | PostgresLoader(connection_string="...", table="users")
+)
+result = asyncio.run(pipeline.run())
+
+# Config-driven: explicit files
+pipeline = Pipeline.from_config_files(
+    extract="configs/extract.yaml",
+    load="configs/load.yaml",
+    variables={"API_KEY": "secret"}
+)
+
+# Config-driven: directory (extract.yaml, transform.yaml, load.yaml)
+pipeline = Pipeline.from_config_dir("pipelines/users/")
+
+# Config-driven: single file
+pipeline = Pipeline.from_config_file("pipelines/users/pipeline.yaml")
+
+result = asyncio.run(pipeline.run())
+```
+
+See **ETL Pipelines** under Core Services for error handling (`error_context`, `ErrorMode`) and variable resolution (`PipelineContext(variables={...})`).
+
 ### Quick Start: Convenience Functions (One-off Use)
 
 ```python
@@ -187,14 +227,29 @@ if result.is_valid:
 
 ### Production Use: Validator Class (Recommended)
 
-For production code with multiple validations, use the `Validator` class for better performance:
+For production code with multiple validations, use the `Validator` class for better performance. Create validators via **factory methods** or from a metadata store:
 
 ```python
 from pycharter import Validator
 
-# Create validator from contract directory or file
-validator = Validator(contract_dir="data/contracts/user")
-# Or: validator = Validator(contract_file="user_contract.yaml")
+# From directory (expects schema.yaml, coercion_rules.yaml, validation_rules.yaml)
+validator = Validator.from_dir("data/contracts/user")
+
+# From explicit files (any filenames)
+validator = Validator.from_files(
+    schema="schemas/user.yaml",
+    coercion_rules="rules/coercion.yaml",
+    validation_rules="rules/validation.yaml"
+)
+
+# From a single contract file
+validator = Validator.from_file("user_contract.yaml")
+
+# From dictionaries
+validator = Validator.from_dict(schema={...}, coercion_rules={...}, validation_rules={...})
+
+# From metadata store (with database)
+validator = Validator(store=store, schema_id="user_schema_v1")
 
 # Validate multiple records efficiently (model is cached)
 result1 = validator.validate({"name": "Alice", "age": 30})
@@ -227,11 +282,12 @@ PyCharter's API is organized into **three tiers** to help you choose the right a
 ### Tier 1: Primary Interfaces (⭐ Recommended for Production)
 
 **Classes** that provide the best performance and most features:
-- **`Validator`** - Primary validation interface (use for multiple validations)
+- **`Validator`** - Primary validation interface (use for multiple validations); create via `from_dir()`, `from_files()`, `from_file()`, `from_dict()` or from store
+- **`Pipeline`** - ETL pipeline (programmatic or config-driven); create via `from_config_files()`, `from_config_dir()`, `from_config_file()` or constructor
 - **`QualityCheck`** - Primary quality assurance interface
 - **`MetadataStoreClient`** - Base class for metadata stores
 
-**When to use**: Production code, batch processing, when you need to validate multiple records.
+**When to use**: Production code, batch processing, when you need to validate multiple records or run ETL pipelines.
 
 ### Tier 2: Convenience Functions (Quick Start)
 
@@ -525,20 +581,24 @@ PyCharter provides validation through three tiers:
 ```python
 from pycharter import Validator, SQLiteMetadataStore
 
-# Option 1: From contract directory (no database)
-validator = Validator(contract_dir="data/contracts/user")
+# Option 1: From directory (schema.yaml, coercion_rules.yaml, validation_rules.yaml)
+validator = Validator.from_dir("data/contracts/user")
+result = validator.validate({"name": "Alice", "age": 30})
+
+# Option 2: From explicit files
+validator = Validator.from_files(schema="schemas/user.yaml", coercion_rules="rules/coercion.yaml")
+result = validator.validate({"name": "Alice", "age": 30})
+
+# Option 3: From single contract file
+validator = Validator.from_file("user_contract.yaml")
 result = validator.validate({"name": "Alice", "age": 30})
 
-# Option 2: From metadata store (with database)
+# Option 4: From metadata store (with database)
 store = SQLiteMetadataStore("metadata.db")
 store.connect()
 validator = Validator(store=store, schema_id="user_schema_v1")
 result = validator.validate({"name": "Alice", "age": 30})
 
-# Option 3: From contract file
-validator = Validator(contract_file="user_contract.yaml")
-result = validator.validate({"name": "Alice", "age": 30})
-
 # Batch validation (efficient - model cached)
 results = validator.validate_batch([data1, data2, data3])
 ```
@@ -579,6 +639,55 @@ results = validate_batch(UserModel, [data1, data2, data3])
 
 ---
 
+### 5b. 🔄 ETL Pipelines (`pycharter.etl_generator`)
+
+**Purpose**: Build and run ETL pipelines programmatically (with the `|` operator) or from YAML configs. No assumptions about project layout—you specify file paths or use a directory with standard filenames.
+
+**When to Use**: When you need to extract, transform, and load data from config-driven or code-defined pipelines (HTTP, files, databases, cloud storage → transforms → Postgres, files, cloud).
+
+**How It Works**:
+- **Programmatic**: `Pipeline(extractor) | transformer | loader`; chain with `|`; call `await pipeline.run()`.
+- **Config-driven**: Load from explicit files (`from_config_files`), from a directory with `extract.yaml`, `transform.yaml`, `load.yaml` (`from_config_dir`), or from a single `pipeline.yaml` (`from_config_file`).
+- **Variables**: Pass `PipelineContext(variables={"API_KEY": "x"})` or `variables={...}` in factory methods; `${VAR}` and `${VAR:-default}` in configs are resolved from these (no built-in `CONTRACT_DIR`).
+- **Async**: `run()` is async; use `asyncio.run(pipeline.run())` in scripts or `await pipeline.run()` in async code.
+- **Error handling**: Optional `error_context` with `ErrorMode` (STRICT, LENIENT, COLLECT) controls whether extraction/load failures raise or are collected in `result.errors`.
+
+**Example**:
+```python
+import asyncio
+from pycharter import Pipeline, PipelineContext, HTTPExtractor, PostgresLoader, Rename, AddField
+
+# Programmatic
+pipeline = (
+    Pipeline(HTTPExtractor(url="https://api.example.com/users"))
+    | Rename({"userName": "name"})
+    | AddField("processed_at", "now()")
+    | PostgresLoader(connection_string="...", table="users")
+)
+result = asyncio.run(pipeline.run())
+
+# Config-driven (explicit files)
+pipeline = Pipeline.from_config_files(
+    extract="configs/extract.yaml",
+    load="configs/load.yaml",
+    variables={"API_KEY": "secret"}
+)
+
+# Config-driven (directory: extract.yaml, transform.yaml, load.yaml)
+pipeline = Pipeline.from_config_dir("pipelines/users/")
+
+# Config-driven (single file)
+pipeline = Pipeline.from_config_file("pipelines/users/pipeline.yaml")
+
+result = asyncio.run(pipeline.run())
+```
+
+**Exceptions**: Pipeline and config loading use PyCharter’s exception hierarchy: `PyCharterError` (base), `ConfigError`, `ConfigValidationError`, `ExpressionError`. See **Exceptions** under API Reference.
+
+**See** `pycharter/etl_generator/ASYNC_AND_EXECUTION.md` for async usage and error modes.
+
+---
+
 ### 6. 🔍 Quality Assurance (`pycharter.quality`)
 
 **Purpose**: Data quality assurance pipeline that polices data according to data contracts, calculates quality metrics, tracks violations, and generates quality reports.
@@ -699,7 +808,7 @@ def process_user_data_quick(raw_data):
 
 ---
 
-### 7. 🌐 REST API (`api/`)
+### 7. 🌐 REST API (`api/`) (`api/`)
 
 **Purpose**: Expose all PyCharter services as REST API endpoints.
 
@@ -747,6 +856,7 @@ See `api/README.md` for complete API documentation.
 | **Pydantic Generator** | JSON Schema | Pydantic models | Storage → Model Generation |
 | **JSON Schema Converter** | Pydantic models | JSON Schema | (Bidirectional) |
 | **Runtime Validator** | Pydantic models + Data | `ValidationResult` | Model Generation → Validation |
+| **ETL Pipelines** | Config files or code | `PipelineResult` | Extract → Transform → Load |
 | **Quality Assurance** | Contract + Data | `QualityReport` | Validation → Quality Monitoring |
 
 Each service is designed to be **independent** yet **composable**, allowing you to use them individually or together as part of a complete data contract management system.
@@ -790,12 +900,9 @@ Product = from_file("product_schema.json", "Product")
 ```python
 from pycharter import Validator
 
-# From contract directory
-validator = Validator(contract_dir="data/contracts/article")
-result = validator.validate({"title": "My Article", "published": True})
-
-# From contract file
-validator = Validator(contract_file="article_contract.yaml")
+# From directory or single file
+validator = Validator.from_dir("data/contracts/article")
+# or: validator = Validator.from_file("article_contract.yaml")
 result = validator.validate({"title": "My Article", "published": True})
 ```
 
@@ -1014,11 +1121,12 @@ PyCharter's API is organized into three tiers to help you choose the right appro
 ```python
 from pycharter import Validator
 
-# Create validator from various sources
-validator = Validator(contract_dir="data/contracts/user")
-validator = Validator(contract_file="contract.yaml")
-validator = Validator(store=store, schema_id="user_schema")
-validator = Validator(contract_dict={...})
+# Create validator via factory methods or store
+validator = Validator.from_dir("data/contracts/user")
+validator = Validator.from_files(schema="schema.yaml", coercion_rules="coercion.yaml")
+validator = Validator.from_file("contract.yaml")
+validator = Validator.from_dict(schema={...}, coercion_rules={...})
+validator = Validator(store=store, schema_id="user_schema")  # from metadata store
 
 # Validate data
 result = validator.validate(data)
@@ -1089,6 +1197,18 @@ store.connect()
 - **`MongoDBMetadataStore(connection_string: str)`** - MongoDB database
 - **`RedisMetadataStore(connection_string: str)`** - Redis database
 
+### Exceptions
+
+PyCharter uses a small exception hierarchy for config and pipeline errors. Catch **`PyCharterError`** to handle any PyCharter failure:
+
+- **`PyCharterError`** - Base for all PyCharter exceptions
+- **`ConfigError`** - Config loading/parsing failures (missing file, invalid YAML)
+- **`ConfigValidationError`** - Schema validation failures (e.g. missing required `type` field)
+- **`ConfigLoadError`** - Config file load errors
+- **`ExpressionError`** - Expression evaluation failures (e.g. invalid syntax in AddField)
+
+Pipeline **`run(error_context=...)`** supports **`ErrorMode`**: **STRICT** (raise on failure), **LENIENT** (log and continue), **COLLECT** (append to `result.errors`). Import from `pycharter.shared.errors`.
+
 ## 🎯 Design Principles & Requirements
 
 Charter is designed to meet the following core requirements: