pycharter 0.0.24__py3-none-any.whl → 0.0.26__py3-none-any.whl

pycharter/metadata_store/README.md

@@ -0,0 +1,229 @@
+# Metadata Store Implementations
+
+PyCharter provides multiple metadata store implementations to suit different use cases:
+
+## Available Implementations
+
+### 1. InMemoryMetadataStore
+**Use case**: Testing, development, prototyping
+
+**Dependencies**: None (built-in)
+
+**Example**:
+```python
+from pycharter import InMemoryMetadataStore
+
+store = InMemoryMetadataStore()
+store.connect()
+
+schema_id = store.store_schema("user", {"type": "object"}, version="1.0")
+schema = store.get_schema(schema_id)
+```
+
+---
+
+### 2. MongoDBMetadataStore
+**Use case**: Document-based storage, flexible schema evolution
+
+**Dependencies**: `pymongo`
+```bash
+pip install pymongo
+```
+
+**Connection string format**: `mongodb://[username:password@]host[:port][/database]`
+
+**Example**:
+```python
+from pycharter import MongoDBMetadataStore
+
+store = MongoDBMetadataStore(
+    connection_string="mongodb://localhost:27017/pycharter",
+    database_name="pycharter"
+)
+store.connect()
+
+schema_id = store.store_schema("user", {"type": "object"}, version="1.0")
+schema = store.get_schema(schema_id)
+```
+
+---
+
+### 3. PostgresMetadataStore
+**Use case**: Relational data, ACID transactions, complex queries
+
+**Dependencies**: `psycopg2-binary`
+```bash
+pip install psycopg2-binary
+```
+
+**Connection string format**: `postgresql://[user[:password]@][host][:port][/database]`
+
+**Example**:
+```python
+from pycharter import PostgresMetadataStore
+
+store = PostgresMetadataStore(
+    connection_string="postgresql://user:password@localhost/pycharter"
+)
+store.connect()
+
+schema_id = store.store_schema("user", {"type": "object"}, version="1.0")
+schema = store.get_schema(schema_id)
+```
+
+**Schema Management**: 
+- Schema is automatically initialized and validated on `connect()`
+- Version tracking ensures schema compatibility
+- Use `store.get_schema_info()` to check schema status
+- See [SCHEMA_MANAGEMENT.md](SCHEMA_MANAGEMENT.md) for details
+
+**CLI Tools**:
+```bash
+# Initialize schema manually
+pycharter db init postgresql://user:pass@localhost/pycharter
+
+# Upgrade to latest version
+pycharter db upgrade postgresql://user:pass@localhost/pycharter
+
+# Check current revision
+pycharter db current postgresql://user:pass@localhost/pycharter
+
+# See migration history
+pycharter db history
+```
+
+---
+
+### 4. SQLiteMetadataStore
+**Use case**: Development, testing, small projects, embedded applications
+
+**Dependencies**: None (sqlite3 is built into Python)
+
+**Connection string format**: `sqlite:///path/to/database.db` or `sqlite:///:memory:`
+
+**Example**:
+```python
+from pycharter import SQLiteMetadataStore
+
+# File-based database
+store = SQLiteMetadataStore("sqlite:///pycharter.db")
+store.connect()
+
+# In-memory database (for testing)
+store = SQLiteMetadataStore("sqlite:///:memory:")
+store.connect()
+
+schema_id = store.store_schema("user", {"type": "object"}, version="1.0")
+schema = store.get_schema(schema_id)
+```
+
+**Schema Management**: 
+- Schema is automatically initialized and validated on `connect()`
+- Uses SQLAlchemy models (same as PostgreSQL)
+- Supports Alembic migrations
+- Use `store.get_schema_info()` to check schema status
+
+**CLI Tools**:
+```bash
+# Initialize schema manually
+pycharter db init sqlite:///pycharter.db
+
+# Run migrations
+pycharter db upgrade sqlite:///pycharter.db
+```
+
+**Notes**:
+- SQLite databases are file-based - easy to backup, share, and version control
+- Perfect for development and testing (no server setup required)
+- Good for small to medium projects
+- Limited concurrent writes (fine for read-heavy workloads)
+- Single-machine only (no network access)
+
+---
+
+### 5. RedisMetadataStore
+**Use case**: High-performance caching, fast lookups, temporary storage
+
+**Dependencies**: `redis`
+```bash
+pip install redis
+```
+
+**Connection string format**: `redis://[password@]host[:port][/database]`
+
+**Example**:
+```python
+from pycharter import RedisMetadataStore
+
+store = RedisMetadataStore(
+    connection_string="redis://localhost:6379/0",
+    key_prefix="pycharter"
+)
+store.connect()
+
+schema_id = store.store_schema("user", {"type": "object"}, version="1.0")
+schema = store.get_schema(schema_id)
+```
+
+---
+
+## Common Usage Pattern
+
+All implementations follow the same interface:
+
+```python
+from pycharter import InMemoryMetadataStore  # or any other implementation
+
+# Initialize and connect
+store = InMemoryMetadataStore()
+store.connect()
+
+# Store schema
+schema_id = store.store_schema(
+    schema_name="user",
+    schema={"type": "object", "properties": {"name": {"type": "string"}}},
+    version="1.0"
+)
+
+# Store ownership and governance rules via metadata
+metadata = {
+    "title": "User Schema",
+    "business_owners": ["data-team@example.com"],
+    "governance_rules": {
+        "pii_encryption": {"type": "encrypt", "fields": ["email"]}
+    }
+}
+store.store_metadata(schema_id, metadata, "schema")
+
+# Retrieve data
+schema = store.get_schema(schema_id)
+retrieved_metadata = store.get_metadata(schema_id, "schema")
+ownership = retrieved_metadata.get("business_owners") if retrieved_metadata else None
+rules = retrieved_metadata.get("governance_rules") if retrieved_metadata else None
+
+# List all schemas
+all_schemas = store.list_schemas()
+
+# Disconnect
+store.disconnect()
+```
+
+## Context Manager Usage
+
+All stores support context manager syntax:
+
+```python
+with InMemoryMetadataStore() as store:
+    schema_id = store.store_schema("user", {"type": "object"})
+    schema = store.get_schema(schema_id)
+    # Automatically disconnects on exit
+```
+
+## Choosing the Right Store
+
+- **InMemoryMetadataStore**: Testing, development, demos (zero dependencies, no persistence)
+- **SQLiteMetadataStore**: Development, testing, small projects (file-based, no server needed)
+- **MongoDBMetadataStore**: Flexible document storage, schema evolution
+- **PostgresMetadataStore**: Production systems, complex queries, ACID guarantees
+- **RedisMetadataStore**: High-performance caching, temporary storage, fast lookups
+

pycharter/quality/README.md

@@ -0,0 +1,235 @@
+# Quality Assurance Module
+
+The Quality Assurance module provides comprehensive data quality checking, metrics calculation, violation tracking, and reporting capabilities for PyCharter.
+
+## Overview
+
+The quality module enables you to:
+- **Run quality checks** against data contracts
+- **Calculate quality metrics** (scores, accuracy, completeness, violation rates)
+- **Track violations** for audit and remediation
+- **Check thresholds** and get alerts when quality degrades
+- **Generate reports** with detailed quality insights
+
+## Core Components
+
+### QualityCheck
+
+The main class for running quality checks. It's orchestrator-agnostic and can be used:
+- Standalone (CLI, API, Python scripts)
+- Within orchestrators (Airflow, Prefect, Dagster)
+- Via API calls
+
+```python
+from pycharter import QualityCheck, QualityCheckOptions
+
+# Create quality check instance
+check = QualityCheck(store=store)  # Optional: pass metadata store
+
+# Run quality check
+report = check.run(
+    schema_id="user_schema_v1",  # Or use contract=...
+    data="data/users.json",  # File path, list, or callable
+    options=QualityCheckOptions(
+        record_violations=True,
+        calculate_metrics=True,
+        check_thresholds=True,
+    )
+)
+
+print(f"Quality Score: {report.quality_score.overall_score:.2f}/100")
+print(f"Passed: {report.passed}")
+```
+
+### QualityMetrics
+
+Calculates quality scores and metrics from validation results:
+
+```python
+from pycharter.quality import QualityMetrics
+
+metrics = QualityMetrics()
+quality_score = metrics.calculate_quality_score(validation_results)
+
+print(f"Overall Score: {quality_score.overall_score}")
+print(f"Accuracy: {quality_score.accuracy:.2%}")
+print(f"Completeness: {quality_score.completeness:.2%}")
+```
+
+### ViolationTracker
+
+Tracks and manages data quality violations:
+
+```python
+from pycharter.quality import ViolationTracker
+
+tracker = ViolationTracker(store=store)
+violation = tracker.record_violation(
+    schema_id="user_schema_v1",
+    record_id="user_123",
+    record_data={"name": "Alice", "age": "invalid"},
+    validation_result=result,
+    severity="warning"
+)
+
+# Query violations
+violations = tracker.get_violations(
+    schema_id="user_schema_v1",
+    status="open"
+)
+```
+
+## Usage Examples
+
+### Standalone Quality Check
+
+```python
+from pycharter import QualityCheck, QualityCheckOptions, QualityThresholds
+
+# Define thresholds
+thresholds = QualityThresholds(
+    min_overall_score=95.0,
+    max_violation_rate=0.05,
+    min_accuracy=0.95
+)
+
+# Run check
+check = QualityCheck()
+report = check.run(
+    contract="contracts/user_contract.yaml",
+    data="data/users.json",
+    options=QualityCheckOptions(
+        check_thresholds=True,
+        thresholds=thresholds
+    )
+)
+
+if not report.passed:
+    print("⚠ Quality check failed!")
+    for breach in report.threshold_breaches:
+        print(f"  - {breach}")
+```
+
+### CLI Usage
+
+```bash
+# Run quality check
+pycharter quality check \
+    --schema-id user_schema_v1 \
+    --data data/users.json \
+    --database-url postgresql://user:pass@localhost/db \
+    --check-thresholds \
+    --output quality_report.json
+
+# Query violations
+pycharter quality violations \
+    --schema-id user_schema_v1 \
+    --status open \
+    --output violations.json
+```
+
+### API Usage
+
+```bash
+# Run quality check via API
+curl -X POST http://localhost:8000/api/v1/quality/check \
+  -H "Content-Type: application/json" \
+  -d '{
+    "schema_id": "user_schema_v1",
+    "data": [{"name": "Alice", "age": 30}],
+    "calculate_metrics": true,
+    "record_violations": true
+  }'
+```
+
+## Quality Metrics
+
+The quality module calculates several metrics:
+
+- **Overall Score** (0-100): Weighted combination of accuracy and completeness
+- **Accuracy** (0-1): Percentage of records that pass validation
+- **Completeness** (0-1): Percentage of required fields present
+- **Violation Rate** (0-1): Percentage of records with violations
+- **Field Scores**: Per-field quality scores
+
+## Thresholds and Alerting
+
+Define quality thresholds to get alerts when quality degrades:
+
+```python
+thresholds = QualityThresholds(
+    min_overall_score=95.0,
+    max_violation_rate=0.05,
+    min_completeness=0.95,
+    min_accuracy=0.95,
+    field_thresholds={
+        "email": {"min_score": 98.0},
+        "phone": {"min_score": 90.0}
+    }
+)
+```
+
+## Integration with Orchestrators
+
+The quality module works seamlessly with orchestrators:
+
+### Airflow
+
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from pycharter import QualityCheck
+
+def quality_check_task():
+    check = QualityCheck(store=store)
+    report = check.run(schema_id="user_schema_v1", data=extract_data())
+    if not report.passed:
+        raise ValueError("Quality check failed")
+
+with DAG("data_pipeline") as dag:
+    quality_check = PythonOperator(
+        task_id="quality_check",
+        python_callable=quality_check_task
+    )
+```
+
+### Prefect
+
+```python
+from prefect import flow, task
+from pycharter import QualityCheck
+
+@task
+def quality_check_task(schema_id: str, data):
+    check = QualityCheck(store=store)
+    return check.run(schema_id=schema_id, data=data)
+
+@flow
+def data_pipeline():
+    data = extract_data()
+    report = quality_check_task("user_schema_v1", data)
+    return report
+```
+
+### Dagster
+
+```python
+from dagster import asset, op
+from pycharter import QualityCheck
+
+@op
+def quality_check_op(context, data):
+    check = QualityCheck(store=store)
+    report = check.run(schema_id="user_schema_v1", data=data)
+    context.log.info(f"Quality score: {report.quality_score.overall_score}")
+    return report
+```
+
+## Future Enhancements
+
+- Database persistence for violations and metrics
+- Scheduled quality monitoring jobs
+- Quality dashboards and visualizations
+- Data profiling capabilities
+- Remediation workflows
+

pycharter/runtime_validator/__init__.py

@@ -62,6 +62,10 @@ from pycharter.runtime_validator.validator import (
     Validator,
     create_validator,
 )
+from pycharter.runtime_validator.utils import (
+    merge_rules_into_schema,
+    get_merged_schema_from_contract,
+)
 
 __all__ = [
     # PRIMARY INTERFACE: Validator and Builder
@@ -85,4 +89,7 @@ __all__ = [
     "validate_input",
     "validate_output",
     "validate_with_contract_decorator",
+    # Schema utilities
+    "merge_rules_into_schema",
+    "get_merged_schema_from_contract",
 ]

pycharter/runtime_validator/utils.py

@@ -69,3 +69,36 @@ def merge_rules_into_schema(
     
     return complete_schema
 
+
+def get_merged_schema_from_contract(contract: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Get merged schema from a contract dictionary.
+    
+    This is a convenience function for cases where you need the merged schema
+    (e.g., for display, documentation generation, or legacy code).
+    
+    For validation, prefer using the Validator class which handles merging
+    internally.
+    
+    Args:
+        contract: Contract dictionary with 'schema', 'coercion_rules', 'validation_rules' keys
+    
+    Returns:
+        Schema dictionary with coercion and validation rules merged into properties
+    
+    Example:
+        >>> contract = build_contract(artifacts)
+        >>> merged = get_merged_schema_from_contract(contract)
+        >>> merged["properties"]["age"]["coercion"]  # Rules are now in schema
+        'coerce_to_integer'
+    
+    Note:
+        The returned schema is a deep copy - modifications won't affect the
+        original contract.
+    """
+    schema = contract.get("schema", {})
+    coercion_rules = contract.get("coercion_rules", {})
+    validation_rules = contract.get("validation_rules", {})
+    
+    return merge_rules_into_schema(schema, coercion_rules, validation_rules)
+

pycharter/runtime_validator/validator.py

@@ -279,8 +279,7 @@ class Validator:
         """
         Load contract from metadata store.
         
-        The complete schema from the store already has rules merged,
-        so we skip the merging step.
+        Fetches raw schema and rules separately, then merges internally.
         
         Args:
             store: MetadataStoreClient instance
@@ -290,17 +289,21 @@ class Validator:
         Raises:
             ValueError: If schema not found in store
         """
-        complete_schema = store.get_complete_schema(schema_id, version)
-        if not complete_schema:
+        # Fetch raw schema (not merged)
+        raw_schema = store.get_schema(schema_id, version)
+        if not raw_schema:
             raise ValueError(f"Schema '{schema_id}' not found in store" + 
                            (f" (version: {version})" if version else ""))
         
-        # The complete_schema already has rules merged by get_complete_schema()
-        # Store it directly - no need to merge again
-        self.schema = complete_schema
-        self.coercion_rules = {}  # Already merged into schema
-        self.validation_rules = {}  # Already merged into schema
-        self._schema_from_store = True  # Flag to skip merging
+        # Fetch rules separately
+        coercion_rules = store.get_coercion_rules(schema_id, version)
+        validation_rules = store.get_validation_rules(schema_id, version)
+        
+        # Store raw components - merge will happen in _merge_rules_into_schema()
+        self.schema = raw_schema
+        self.coercion_rules = self._extract_rules(coercion_rules) if coercion_rules else {}
+        self.validation_rules = self._extract_rules(validation_rules) if validation_rules else {}
+        self._schema_from_store = False  # Let internal merge happen
     
     def _load_from_metadata(self, metadata: ContractMetadata) -> None:
         """

pycharter/ui/.eslintrc.json

@@ -0,0 +1,4 @@
+{
+  "extends": "next/core-web-vitals"
+}
+