pycharter 0.0.20__tar.gz → 0.0.23__tar.gz

{pycharter-0.0.20 → pycharter-0.0.23}/MANIFEST.in

@@ -17,8 +17,10 @@ prune ui/out
 prune ui/src
 prune ui/public
 # Exclude source files but NOT static build files
+# Note: ui/static is explicitly included above, so these excludes won't affect it
 recursive-exclude ui/src *.ts *.tsx *.js *.json *.md
 recursive-exclude ui/src *.config.*
+# Exclude other UI files (but not ui/static which is already included)
 recursive-exclude ui *.ts *.tsx *.json *.md
 recursive-exclude ui *.config.*
 

{pycharter-0.0.20/pycharter.egg-info → pycharter-0.0.23}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pycharter
-Version: 0.0.20
+Version: 0.0.23
 Summary: A Python package for data contract management with five core services: contract parsing, metadata storage, Pydantic generation, JSON Schema conversion, and runtime validation
 Author-email: semantic developers <na@example.com>
 License-Expression: MIT
@@ -25,6 +25,7 @@ Requires-Dist: numpy>=1.24.0
 Requires-Dist: pyyaml>=6.0.0
 Requires-Dist: sqlalchemy>=2.0.0
 Requires-Dist: alembic>=1.13.0
+Requires-Dist: jsonata-python>=0.6.0
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
 Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
@@ -53,6 +54,11 @@ Requires-Dist: pyspark>=3.5.0; extra == "worker"
 Requires-Dist: redis>=5.0.0; extra == "worker"
 Provides-Extra: etl
 Requires-Dist: sshtunnel>=0.4.0; extra == "etl"
+Requires-Dist: boto3>=1.26.0; extra == "etl"
+Requires-Dist: google-cloud-storage>=2.0.0; extra == "etl"
+Requires-Dist: azure-storage-blob>=12.0.0; extra == "etl"
+Requires-Dist: openpyxl>=3.0.0; extra == "etl"
+Requires-Dist: lxml>=4.9.0; extra == "etl"
 Dynamic: license-file
 
 # PyCharter
@@ -158,12 +164,15 @@ pycharter ui dev     # Development mode with hot reload
 
 ## 🚀 Quick Start
 
+### Quick Start: Convenience Functions (One-off Use)
+
 ```python
-from pycharter import from_dict
+from pycharter import from_dict, validate
 
 # Define your JSON schema
 schema = {
     "type": "object",
+    "version": "1.0.0",
     "properties": {
         "name": {"type": "string"},
         "age": {"type": "integer"},
@@ -172,15 +181,91 @@ schema = {
     "required": ["name", "age"]
 }
 
-# Generate a Pydantic model
+# Generate a Pydantic model (convenience function)
 Person = from_dict(schema, "Person")
 
-# Use it like any Pydantic model
-person = Person(name="Alice", age=30, email="alice@example.com")
-print(person.name)  # Output: Alice
-print(person.age)   # Output: 30
+# Validate data
+result = validate(Person, {"name": "Alice", "age": 30, "email": "alice@example.com"})
+if result.is_valid:
+    print(f"Valid: {result.data.name}")  # Output: Valid: Alice
+```
+
+### Production Use: Validator Class (Recommended)
+
+For production code with multiple validations, use the `Validator` class for better performance:
+
+```python
+from pycharter import Validator
+
+# Create validator from contract directory or file
+validator = Validator(contract_dir="data/contracts/user")
+# Or: validator = Validator(contract_file="user_contract.yaml")
+
+# Validate multiple records efficiently (model is cached)
+result1 = validator.validate({"name": "Alice", "age": 30})
+result2 = validator.validate({"name": "Bob", "age": 25})
+
+# Batch validation
+results = validator.validate_batch([data1, data2, data3])
 ```
 
+### With Metadata Store
+
+```python
+from pycharter import Validator, SQLiteMetadataStore
+
+# Connect to metadata store
+store = SQLiteMetadataStore("metadata.db")
+store.connect()
+
+# Create validator from store
+validator = Validator(store=store, schema_id="user_schema_v1")
+
+# Validate data
+result = validator.validate({"name": "Alice", "age": 30})
+```
+
+## 📐 API Organization
+
+PyCharter's API is organized into **three tiers** to help you choose the right approach for your use case:
+
+### Tier 1: Primary Interfaces (⭐ Recommended for Production)
+
+**Classes** that provide the best performance and most features:
+- **`Validator`** - Primary validation interface (use for multiple validations)
+- **`QualityCheck`** - Primary quality assurance interface
+- **`MetadataStoreClient`** - Base class for metadata stores
+
+**When to use**: Production code, batch processing, when you need to validate multiple records.
+
+### Tier 2: Convenience Functions (Quick Start)
+
+**Functions** that make common tasks easy and discoverable:
+- **Input helpers**: `from_dict()`, `from_file()`, `from_json()`, `from_url()`
+- **Output helpers**: `to_dict()`, `to_file()`, `to_json()`
+- **Validation helpers**: `validate_with_store()`, `validate_with_contract()`
+- **Contract helpers**: `parse_contract_file()`, `build_contract()`
+
+**When to use**: Quick scripts, one-off validations, exploratory work, learning the library.
+
+### Tier 3: Low-Level Utilities
+
+**Functions** for when you already have models or need fine-grained control:
+- **`validate()`** - Validate with existing Pydantic model
+- **`validate_batch()`** - Batch validate with existing model
+- **`model_to_schema()`** - Core conversion function
+
+**When to use**: Advanced use cases, when you've already generated models, custom workflows.
+
+### Choosing the Right Approach
+
+| Use Case | Recommended Approach | Example |
+|----------|---------------------|---------|
+| **Production pipeline with multiple validations** | `Validator` class | `validator = Validator(store=store, schema_id="schema"); validator.validate(data)` |
+| **Quick one-off validation** | Convenience function | `validate_with_contract("contract.yaml", data)` |
+| **You already have a model** | Low-level function | `validate(UserModel, data)` |
+| **Batch processing** | `Validator.validate_batch()` | `validator.validate_batch([data1, data2, data3])` |
+
 ## 🏗️ Core Services & Data Production Journey
 
 PyCharter provides **eight core services** that work together to support a complete data production journey, from contract specification to quality assurance. Each service plays a critical role in managing data contracts and ensuring data quality throughout your pipeline.
@@ -291,19 +376,20 @@ result = validate_with_contract(contract, {"name": "Alice", "age": "30"})
 
 **Available Implementations**:
 - **PostgresMetadataStore** - For PostgreSQL databases (recommended for production)
+- **SQLiteMetadataStore** - For SQLite databases (great for development and small deployments)
 - **MongoDBMetadataStore** - For MongoDB databases
 - **RedisMetadataStore** - For Redis databases
 - **InMemoryMetadataStore** - For testing and development (no persistence)
 
 **Example**:
 ```python
-from pycharter import PostgresMetadataStore, parse_contract_file
+from pycharter import SQLiteMetadataStore, parse_contract_file
 
 # Parse contract
 metadata = parse_contract_file("contract.yaml")
 
-# Use PostgreSQL metadata store (or MongoDBMetadataStore, RedisMetadataStore, etc.)
-store = PostgresMetadataStore(connection_string="postgresql://user:pass@localhost:5432/pycharter")
+# Use SQLite metadata store (or PostgresMetadataStore, MongoDBMetadataStore, RedisMetadataStore, etc.)
+store = SQLiteMetadataStore("metadata.db")
 store.connect()
 
 # Store decomposed components
@@ -415,6 +501,24 @@ ProductModel = from_dict(schema, "Product")  # Round-trip
 
 **When to Use**: In your data processing scripts, ETL pipelines, API endpoints, or any place where you need to validate incoming data against contract specifications.
 
+**API Organization**:
+
+PyCharter provides validation through three tiers:
+
+1. **Tier 1: Validator Class** (⭐ **PRIMARY INTERFACE** - Recommended for production)
+   - Best performance for multiple validations (model is cached)
+   - Supports all data sources (contract files, directories, stores, dictionaries)
+   - Reusable instance for batch processing
+
+2. **Tier 2: Convenience Functions** (Quick start - one-off validations)
+   - `validate_with_store()` - Quick validation with metadata store
+   - `validate_with_contract()` - Quick validation with contract file/dict
+   - `get_model_from_store()` / `get_model_from_contract()` - Get model for reuse
+
+3. **Tier 3: Low-Level Functions** (When you already have a model)
+   - `validate()` - Validate single record with existing model
+   - `validate_batch()` - Batch validate with existing model
+
 **How It Works**:
 - Takes a Pydantic model (generated from a schema) and raw data
 - Validates data against the model's constraints
@@ -422,55 +526,60 @@ ProductModel = from_dict(schema, "Product")  # Round-trip
 - Supports single record and batch validation
 - Can be used in strict mode (raises exceptions) or lenient mode (returns results)
 
-**Two Validation Modes**:
+**Example - Validator Class (Recommended)**:
+```python
+from pycharter import Validator, SQLiteMetadataStore
 
-1. **Database-Backed Validation** (with metadata store):
-   - Retrieve schemas and rules from database
-   - Use `validate_with_store()`, `validate_batch_with_store()`, `get_model_from_store()`
+# Option 1: From contract directory (no database)
+validator = Validator(contract_dir="data/contracts/user")
+result = validator.validate({"name": "Alice", "age": 30})
 
-2. **Contract-Based Validation** (no database required):
-   - Validate directly against contract files or dictionaries
-   - Use `validate_with_contract()`, `validate_batch_with_contract()`, `get_model_from_contract()`
+# Option 2: From metadata store (with database)
+store = SQLiteMetadataStore("metadata.db")
+store.connect()
+validator = Validator(store=store, schema_id="user_schema_v1")
+result = validator.validate({"name": "Alice", "age": 30})
 
-**Example - Database-Backed**:
+# Option 3: From contract file
+validator = Validator(contract_file="user_contract.yaml")
+result = validator.validate({"name": "Alice", "age": 30})
+
+# Batch validation (efficient - model cached)
+results = validator.validate_batch([data1, data2, data3])
+```
+
+**Example - Convenience Functions (Quick Start)**:
 ```python
-from pycharter import validate_with_store, InMemoryMetadataStore
+from pycharter import validate_with_store, validate_with_contract, SQLiteMetadataStore
 
-# Store and validate with database
-store = InMemoryMetadataStore()
+# Quick validation with store
+store = SQLiteMetadataStore("metadata.db")
 store.connect()
-# ... store schema, rules, etc. ...
-
-# Validate using store
 result = validate_with_store(store, "user_schema_v1", {"name": "Alice", "age": 30})
-if result.is_valid:
-    print(f"Valid user: {result.data.name}")
+
+# Quick validation with contract file (no database)
+result = validate_with_contract("user_contract.yaml", {"name": "Alice", "age": 30})
 ```
 
-**Example - Contract-Based (No Database)**:
+**Example - Low-Level (When You Have a Model)**:
 ```python
-from pycharter import validate_with_contract, get_model_from_contract, validate
+from pycharter import from_dict, validate, validate_batch
 
-# Validate directly from contract file (simplest)
-result = validate_with_contract(
-    "data/examples/book/book_contract.yaml",
-    {"isbn": "1234567890", "title": "Book", ...}
-)
+# Generate model
+UserModel = from_dict(schema, "User")
 
-# Or get model once, validate multiple times (efficient)
-BookModel = get_model_from_contract("book_contract.yaml")
-result1 = validate(BookModel, data1)
-result2 = validate(BookModel, data2)
+# Validate single record
+result = validate(UserModel, {"name": "Alice", "age": 30})
 
-# Or from dictionary
-contract = {
-    "schema": {"type": "object", "properties": {...}},
-    "coercion_rules": {"rules": {...}},
-    "validation_rules": {"rules": {...}}
-}
-result = validate_with_contract(contract, data)
+# Batch validate
+results = validate_batch(UserModel, [data1, data2, data3])
 ```
 
+**Performance Tips**:
+- ⚡ **For multiple validations**: Use `Validator` class (model is cached)
+- ⚡ **For one-off validations**: Convenience functions are fine
+- ⚡ **For batch processing**: Use `Validator.validate_batch()` or `validate_batch()`
+
 **Contribution to Journey**: The runtime validator is the **enforcement layer** that ensures data quality in production. It validates actual data against contract specifications, catching violations early and preventing bad data from propagating through your systems. It supports both database-backed workflows (for production systems with metadata stores) and contract-based workflows (for simpler use cases without database dependencies).
 
 ---
@@ -533,9 +642,9 @@ Here's how all services work together in a complete data production journey:
 ```python
 from pycharter import (
     parse_contract_file,
-    PostgresMetadataStore,
+    SQLiteMetadataStore,
     from_dict,
-    validate,
+    Validator,
     to_dict
 )
 
@@ -543,7 +652,7 @@ from pycharter import (
 metadata = parse_contract_file("user_contract.yaml")
 
 # Step 2: Store metadata in database
-store = PostgresMetadataStore(connection_string="postgresql://user:pass@localhost:5432/pycharter")
+store = SQLiteMetadataStore("metadata.db")
 store.connect()
 schema_id = store.store_schema("user", metadata.schema, version="1.0")
 
@@ -570,14 +679,27 @@ UserModel = from_dict(schema, "User")
 schema_doc = to_dict(UserModel)
 
 # Step 5: Validate data in production pipeline
+# Option A: Using Validator class (recommended for production)
+validator = Validator(store=store, schema_id=schema_id)
+
 def process_user_data(raw_data):
-    result = validate(UserModel, raw_data)
+    result = validator.validate(raw_data)
     if result.is_valid:
         # Process validated data
         return result.data
     else:
         # Handle validation errors
         raise ValueError(f"Invalid data: {result.errors}")
+
+# Option B: Using convenience function (quick start)
+from pycharter import validate_with_store
+
+def process_user_data_quick(raw_data):
+    result = validate_with_store(store, schema_id, raw_data)
+    if result.is_valid:
+        return result.data
+    else:
+        raise ValueError(f"Invalid data: {result.errors}")
 ```
 
 ---
@@ -646,12 +768,14 @@ Each service is designed to be **independent** yet **composable**, allowing you
 
 ### Basic Usage
 
+**Using Convenience Functions** (Quick Start):
 ```python
 from pycharter import from_dict, from_json, from_file
 
 # From dictionary
 schema = {
     "type": "object",
+    "version": "1.0.0",
     "properties": {
         "title": {"type": "string"},
         "published": {"type": "boolean", "default": False}
@@ -660,13 +784,26 @@ schema = {
 Article = from_dict(schema, "Article")
 
 # From JSON string
-schema_json = '{"type": "object", "properties": {"name": {"type": "string"}}}'
+schema_json = '{"type": "object", "version": "1.0.0", "properties": {"name": {"type": "string"}}}'
 User = from_json(schema_json, "User")
 
 # From file
 Product = from_file("product_schema.json", "Product")
 ```
 
+**Using Validator Class** (Production):
+```python
+from pycharter import Validator
+
+# From contract directory
+validator = Validator(contract_dir="data/contracts/article")
+result = validator.validate({"title": "My Article", "published": True})
+
+# From contract file
+validator = Validator(contract_file="article_contract.yaml")
+result = validator.validate({"title": "My Article", "published": True})
+```
+
 ### Nested Objects
 
 ```python
@@ -674,6 +811,7 @@ from pycharter import from_dict
 
 schema = {
     "type": "object",
+    "version": "1.0.0",
     "properties": {
         "name": {"type": "string"},
         "address": {
@@ -707,6 +845,7 @@ from pycharter import from_dict
 
 schema = {
     "type": "object",
+    "version": "1.0.0",
     "properties": {
         "tags": {
             "type": "array",
@@ -739,13 +878,14 @@ print(cart.items[0].name)  # Output: Apple
 
 ### Coercion and Validation
 
-Charter supports **coercion** (pre-validation transformation) and **validation** (post-validation checks):
+PyCharter supports **coercion** (pre-validation transformation) and **validation** (post-validation checks):
 
 ```python
 from pycharter import from_dict
 
 schema = {
     "type": "object",
+    "version": "1.0.0",
     "properties": {
         "flight_number": {
             "type": "integer",
@@ -871,13 +1011,88 @@ register_validation("must_be_positive", must_be_positive)
 
 ## 📖 API Reference
 
-### Main Functions
+PyCharter's API is organized into three tiers to help you choose the right approach:
 
+### Tier 1: Primary Interfaces (Classes - Best Performance)
+
+**Validator** - Primary validation interface (recommended for production)
+```python
+from pycharter import Validator
+
+# Create validator from various sources
+validator = Validator(contract_dir="data/contracts/user")
+validator = Validator(contract_file="contract.yaml")
+validator = Validator(store=store, schema_id="user_schema")
+validator = Validator(contract_dict={...})
+
+# Validate data
+result = validator.validate(data)
+results = validator.validate_batch([data1, data2])
+model = validator.get_model()  # Get the generated Pydantic model
+```
+
+**QualityCheck** - Primary quality assurance interface
+```python
+from pycharter import QualityCheck, QualityCheckOptions
+
+check = QualityCheck(store=store)
+report = check.run(schema_id="user_schema", data=data, options=QualityCheckOptions(...))
+```
+
+**MetadataStoreClient** - Base class for metadata stores
+```python
+from pycharter import MetadataStoreClient, SQLiteMetadataStore, PostgresMetadataStore
+
+store = SQLiteMetadataStore("metadata.db")
+store.connect()
+```
+
+### Tier 2: Convenience Functions (Quick Start)
+
+**Pydantic Generator** - Input type helpers
 - **`from_dict(schema: dict, model_name: str = "DynamicModel")`** - Create model from dictionary
 - **`from_json(json_string: str, model_name: str = "DynamicModel")`** - Create model from JSON string
-- **`from_file(file_path: str, model_name: str = None)`** - Create model from JSON file
+- **`from_file(file_path: str, model_name: str = None)`** - Create model from file (JSON/YAML)
 - **`from_url(url: str, model_name: str = "DynamicModel")`** - Create model from URL
-- **`schema_to_model(schema: dict, model_name: str = "DynamicModel")`** - Low-level model generator
+- **`generate_model(schema: dict, model_name: str = "DynamicModel")`** - Advanced: more control
+- **`generate_model_file(schema: dict, output_path: str, model_name: str = "DynamicModel")`** - Generate and save to file
+
+**JSON Schema Converter** - Output type helpers
+- **`to_dict(model: Type[BaseModel], ...)`** - Convert model to JSON Schema dictionary
+- **`to_file(model: Type[BaseModel], file_path: str, ...)`** - Convert model to file
+- **`to_json(model: Type[BaseModel], ...)`** - Convert model to JSON string
+- **`model_to_schema(model: Type[BaseModel], ...)`** - Advanced: core conversion function
+
+**Runtime Validator** - Data source helpers
+- **`validate_with_store(store, schema_id, data, ...)`** - Quick validation with metadata store
+- **`validate_batch_with_store(store, schema_id, data_list, ...)`** - Batch validation with store
+- **`validate_with_contract(contract, data, ...)`** - Quick validation with contract file/dict
+- **`validate_batch_with_contract(contract, data_list, ...)`** - Batch validation with contract
+- **`get_model_from_store(store, schema_id, ...)`** - Get model from metadata store
+- **`get_model_from_contract(contract, ...)`** - Get model from contract
+- **`validate_input(contract, ...)`** - Decorator for function input validation
+- **`validate_output(contract, ...)`** - Decorator for function output validation
+- **`validate_with_contract_decorator(contract, ...)`** - Decorator for contract-based validation
+
+**Contract Management**
+- **`parse_contract(contract_dict: dict)`** - Parse contract dictionary
+- **`parse_contract_file(file_path: str)`** - Parse contract file (YAML/JSON)
+- **`build_contract(artifacts: ContractArtifacts)`** - Build contract from artifacts
+- **`build_contract_from_store(store, schema_id, ...)`** - Build contract from metadata store
+
+### Tier 3: Low-Level Utilities (When You Have Models)
+
+- **`validate(model: Type[BaseModel], data: dict, strict: bool = False)`** - Validate single record
+- **`validate_batch(model: Type[BaseModel], data_list: List[dict], strict: bool = False)`** - Batch validate
+- **`ValidationResult`** - Result class with `is_valid`, `data`, and `errors` attributes
+
+### Metadata Store Implementations
+
+- **`InMemoryMetadataStore()`** - In-memory store (testing/development)
+- **`SQLiteMetadataStore(database_path: str)`** - SQLite database
+- **`PostgresMetadataStore(connection_string: str)`** - PostgreSQL database
+- **`MongoDBMetadataStore(connection_string: str)`** - MongoDB database
+- **`RedisMetadataStore(connection_string: str)`** - Redis database
 
 ## 🎯 Design Principles & Requirements
 
@@ -979,21 +1194,33 @@ pytest -k "coercion"
 
 ## 📦 Publishing to PyPI
 
+**Automatic publishing via GitHub Releases (Trusted Publishing - no tokens needed!):**
+
 ```bash
-# Update version in pyproject.toml
-# Clean previous builds
-make clean
+# 1. Update version in pyproject.toml
+# version = "0.0.21"
+
+# 2. Commit and push
+git add pyproject.toml
+git commit -m "Bump version to 0.0.21"
+git push
 
-# Build package
-make build
+# 3. Create GitHub Release (automatically publishes to PyPI)
+gh release create v0.0.21 --title "v0.0.21" --notes "Release notes"
+```
 
-# Test on TestPyPI
-make publish-test
+The workflow automatically:
+- ✅ Builds UI
+- ✅ Builds Python package
+- ✅ Publishes to PyPI (using Trusted Publishing)
 
-# Publish to PyPI
-make publish
+**Local build:**
+```bash
+make build  # Builds package (UI built automatically via setup.py)
 ```
 
+See `PUBLISHING.md` for complete documentation.
+
 ## 📋 JSON Schema Compliance
 
 PyCharter is fully compliant with **JSON Schema Draft 2020-12** standard: