ferrox-py-utils 1.0.0__tar.gz

ferrox_py_utils-1.0.0/.gitignore

@@ -0,0 +1,5 @@
+venv/
+__pycache__/
+*.pyc
+.env
+.pytest_cache/

ferrox_py_utils-1.0.0/PKG-INFO

@@ -0,0 +1,67 @@
+Metadata-Version: 2.5
+Name: ferrox-py-utils
+Version: 1.0.0
+Summary: Data Engineering and ETL utilities for the Ferrox ecosystem.
+Author: AI-Autistic-Intelligence
+Requires-Python: >=3.11
+Requires-Dist: ferrox-py>=1.0.0
+Requires-Dist: pydantic>=2.0
+Description-Content-Type: text/markdown
+
+# 🛠️ Ferrox-Py-Utils (Data Engineering)
+
+## 1. Overview (What does this do?)
+The `ferrox-py-utils` package is a specialized extension of the Ferrox ecosystem dedicated to data manipulation, ETL (Extract, Transform, Load) pipelines, and cross-platform data movement. It provides agnostic `Connectors` (e.g., for AWS S3, CSV files, or REST APIs) and a `PipelineOrchestrator` to seamlessly sequence data transformation jobs without writing monolithic scripts.
+
+## 2. Philosophy (Why does it exist?)
+Data engineering often suffers from "wild copy-pasting" where scripts to import users or export CSVs are hastily written and tightly coupled to specific database schemas or cloud vendors. The philosophy here is absolute **agnosticism**. Instead of building monolithic ETL scripts, this package enforces a modular approach where small, isolated tasks are injected into an orchestrator. This allows data engineers to reuse extraction and validation logic across entirely different projects.
+
+## 3. Target Audience (Who is it for?)
+This package is built for Data Engineers and backend developers who need to quickly stand up a Data Platform for ingestion, parsing, and bulk loading of large datasets (like CSV or JSON) into Data Lakes or Object Storage (such as Amazon S3, MinIO, or Google Cloud Storage).
+
+## 4. Architecture (How does it work?)
+The data architecture rests on three pillars:
+- **Connectors**: Classes inheriting from an abstract `BaseConnector` to standardize streaming I/O operations (Read/Write/Delete), entirely isolating the pipeline from the specific storage vendor.
+- **PipelineOrchestrator**: A linear execution engine that passes a shared state (`context`) through a sequence of nodes (steps), ensuring proper error isolation and retry mechanics.
+- **Schema Registry**: Native integration with Pydantic to register and enforce strict validation on datasets in transit, ensuring corrupted data never enters the database.
+
+## 5. Installation / Setup
+Ensure you are using Python 3.11+. The package installs basic dependencies, but you may need to install specific data drivers (like `boto3` or `pandas`) depending on the connectors you intend to use.
+
+```bash
+pip install ferrox-py-utils
+# Optional extensions:
+# pip install boto3 pandas
+```
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py_utils.connectors.csv import CsvConnector
+from ferrox_py_utils.pipelines.orchestrator import PipelineOrchestrator
+
+# 1. Setup the agnostic connector
+csv_connector = CsvConnector(file_path="/tmp/data.csv")
+
+# 2. Define isolated pipeline steps
+def step_read_csv(ctx):
+    ctx['data'] = csv_connector.read()
+    return ctx
+
+def step_transform(ctx):
+    # Transform logic here...
+    ctx['data'] = [row for row in ctx['data'] if row.get("active")]
+    return ctx
+
+# 3. Orchestrate and execute
+orchestrator = PipelineOrchestrator()
+orchestrator.add_step("Read Data", step_read_csv)
+orchestrator.add_step("Clean Data", step_transform)
+
+final_context = orchestrator.execute()
+print(f"Processed {len(final_context['data'])} records.")
+```
+
+## 7. Ecosystem Integration
+This module is fully integrated with the core `ferrox-py` ecosystem:
+- **Core (IoC Container)**: The framework's Dependency Injection container is used to instantiate Connectors globally as Singletons, meaning you only initialize your S3 credentials once.
+- **AuthModule (ferrox-py-auth)**: RBAC can be utilized to restrict which users or system roles are authorized to trigger specific data pipelines via the API Gateway.

ferrox_py_utils-1.0.0/README.md

@@ -0,0 +1,57 @@
+# 🛠️ Ferrox-Py-Utils (Data Engineering)
+
+## 1. Overview (What does this do?)
+The `ferrox-py-utils` package is a specialized extension of the Ferrox ecosystem dedicated to data manipulation, ETL (Extract, Transform, Load) pipelines, and cross-platform data movement. It provides agnostic `Connectors` (e.g., for AWS S3, CSV files, or REST APIs) and a `PipelineOrchestrator` to seamlessly sequence data transformation jobs without writing monolithic scripts.
+
+## 2. Philosophy (Why does it exist?)
+Data engineering often suffers from "wild copy-pasting" where scripts to import users or export CSVs are hastily written and tightly coupled to specific database schemas or cloud vendors. The philosophy here is absolute **agnosticism**. Instead of building monolithic ETL scripts, this package enforces a modular approach where small, isolated tasks are injected into an orchestrator. This allows data engineers to reuse extraction and validation logic across entirely different projects.
+
+## 3. Target Audience (Who is it for?)
+This package is built for Data Engineers and backend developers who need to quickly stand up a Data Platform for ingestion, parsing, and bulk loading of large datasets (like CSV or JSON) into Data Lakes or Object Storage (such as Amazon S3, MinIO, or Google Cloud Storage).
+
+## 4. Architecture (How does it work?)
+The data architecture rests on three pillars:
+- **Connectors**: Classes inheriting from an abstract `BaseConnector` to standardize streaming I/O operations (Read/Write/Delete), entirely isolating the pipeline from the specific storage vendor.
+- **PipelineOrchestrator**: A linear execution engine that passes a shared state (`context`) through a sequence of nodes (steps), ensuring proper error isolation and retry mechanics.
+- **Schema Registry**: Native integration with Pydantic to register and enforce strict validation on datasets in transit, ensuring corrupted data never enters the database.
+
+## 5. Installation / Setup
+Ensure you are using Python 3.11+. The package installs basic dependencies, but you may need to install specific data drivers (like `boto3` or `pandas`) depending on the connectors you intend to use.
+
+```bash
+pip install ferrox-py-utils
+# Optional extensions:
+# pip install boto3 pandas
+```
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py_utils.connectors.csv import CsvConnector
+from ferrox_py_utils.pipelines.orchestrator import PipelineOrchestrator
+
+# 1. Setup the agnostic connector
+csv_connector = CsvConnector(file_path="/tmp/data.csv")
+
+# 2. Define isolated pipeline steps
+def step_read_csv(ctx):
+    ctx['data'] = csv_connector.read()
+    return ctx
+
+def step_transform(ctx):
+    # Transform logic here...
+    ctx['data'] = [row for row in ctx['data'] if row.get("active")]
+    return ctx
+
+# 3. Orchestrate and execute
+orchestrator = PipelineOrchestrator()
+orchestrator.add_step("Read Data", step_read_csv)
+orchestrator.add_step("Clean Data", step_transform)
+
+final_context = orchestrator.execute()
+print(f"Processed {len(final_context['data'])} records.")
+```
+
+## 7. Ecosystem Integration
+This module is fully integrated with the core `ferrox-py` ecosystem:
+- **Core (IoC Container)**: The framework's Dependency Injection container is used to instantiate Connectors globally as Singletons, meaning you only initialize your S3 credentials once.
+- **AuthModule (ferrox-py-auth)**: RBAC can be utilized to restrict which users or system roles are authorized to trigger specific data pipelines via the API Gateway.

ferrox_py_utils-1.0.0/docs/connectors.md

@@ -0,0 +1,50 @@
+# Connectors
+
+## 1. Overview (What does this do?)
+The `connectors` module standardizes read and write access to external data sources. It provides a set of pre-built, reusable classes that can interact with Object Storage (S3, MinIO), local file systems, and tabular formats (CSV, Excel) using a unified programming interface.
+
+## 2. Philosophy (Why does it exist?)
+Interacting with external data sources usually involves writing boilerplate code using varying SDKs (e.g., `boto3` for S3, native `open()` for files, `requests` for APIs). This creates tightly coupled code that is difficult to test and impossible to swap out easily. By forcing all data I/O through a generic `BaseConnector` interface, the business logic remains entirely oblivious to where the data is actually coming from or going to, allowing developers to easily mock these interactions during unit testing.
+
+## 3. Target Audience (Who is it for?)
+This module is intended for Data Engineers writing ETL pipelines or backend developers who need to generate, upload, or process reports, backups, and user-uploaded files without wrestling with raw provider SDKs.
+
+## 4. Architecture (How does it work?)
+- **BaseConnector**: An abstract base class defining the strict I/O contract (`read` and `write` methods).
+- **S3Connector**: Facilitates interaction with S3-compatible object storage (AWS, MinIO). It handles authentication securely and manages streaming byte uploads and downloads to prevent memory exhaustion on large files.
+- **CsvConnector**: Simplifies handling CSV files, natively supporting chunking mechanisms (often wrapping Pandas or the native Python `csv` module) to process massive datasets sequentially.
+
+## 5. Installation / Setup
+The `BaseConnector` requires no external libraries. However, to use the specific implementations, you must install the respective drivers.
+
+```bash
+# For S3 connectivity
+pip install boto3
+
+# For advanced CSV/Excel manipulation
+pip install pandas
+```
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py_utils.connectors.s3 import S3Connector
+
+# 1. Initialize the connector (usually done within the IoC Container)
+s3_connector = S3Connector(
+    endpoint_url="http://localhost:9000",
+    access_key="minioadmin",
+    secret_key="minioadmin",
+    bucket_name="bronze-layer"
+)
+
+# 2. Use the unified interface
+def backup_database_dump(data_bytes: bytes):
+    # The business logic only calls 'write'
+    s3_connector.write(file_path="backups/db_dump.sql", data=data_bytes)
+
+# 3. Read data back
+raw_data = s3_connector.read(file_path="backups/db_dump.sql")
+```
+
+## 7. Ecosystem Integration
+Connectors are explicitly designed to be registered as **Providers** in the core `ferrox-py` Dependency Injection Container. This ensures that the application only maintains one active connection pool to S3 or a remote FTP server, injecting it seamlessly into the **PipelineOrchestrator** steps when needed.

ferrox_py_utils-1.0.0/docs/overview.md

@@ -0,0 +1,43 @@
+# Ferrox-Py-Utils Overview
+
+## 1. Overview (What does this do?)
+The `ferrox-py-utils` package implements a decoupled data architecture based on interchangeable connectors and a linear execution pipeline. It provides the scaffolding required to build robust Data Platforms entirely within the Ferrox ecosystem.
+
+## 2. Philosophy (Why does it exist?)
+Traditional ETL (Extract, Transform, Load) scripts are often written as massive, procedural files (e.g., `import_users.py`). These scripts are incredibly fragile, hard to test, and impossible to reuse. This package mandates a functional, "matryoshka" approach: extraction is isolated to Connectors, transformation is isolated to pure functions (Steps), and execution is managed by an Orchestrator. This ensures that a bug in one transformation step doesn't crash the entire data ingestion process without proper logging.
+
+## 3. Target Audience (Who is it for?)
+This overview is for Software Architects and Data Engineers who are designing the data ingestion, processing, and warehousing layers of their application and need a structured, maintainable framework rather than a collection of disorganized Python scripts.
+
+## 4. Architecture (How does it work?)
+The data engineering process is strictly divided into three phases:
+1. **Isolated Extraction (I/O)**: Data is read from external sources (S3, Databases, APIs) exclusively through specialized *Connectors*.
+2. **In-Memory Transformation**: A *PipelineOrchestrator* executes a sequence of functional steps, mutating a shared state object (`context`) in a predictable order.
+3. **Guaranteed Validation**: The *Schema Registry* ensures that the data being transformed strictly adheres to predefined contracts (DTOs) before it is passed to the next step or loaded into the final Database (the Bronze/Silver/Gold layers).
+
+## 5. Installation / Setup
+The package is installed via standard pip mechanics. It relies heavily on the core `ferrox-py` framework for architectural consistency.
+
+```bash
+pip install ferrox-py-utils
+```
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py.core.container import Container
+from ferrox_py_utils.connectors.s3 import S3Connector
+
+# Centralized setup of data dependencies
+container = Container()
+container.register("s3_connector", S3Connector(
+    endpoint_url="https://s3.amazonaws.com",
+    access_key="...",
+    secret_key="...",
+    bucket_name="my-data-lake"
+))
+
+# The connector is now available globally for any Pipeline to use
+```
+
+## 7. Ecosystem Integration
+The entire philosophy of this package is built upon the **Inversion of Control (IoC) Container** provided by the `ferrox-py` core. Connectors and Pipeline steps are resolved dynamically via Dependency Injection, ensuring that unit testing an ETL pipeline doesn't require spinning up an actual AWS environment.

ferrox_py_utils-1.0.0/docs/pipelines.md

@@ -0,0 +1,45 @@
+# Pipelines
+
+## 1. Overview (What does this do?)
+The `PipelineOrchestrator` is the execution engine at the heart of the `ferrox-py-utils` data engineering toolkit. It provides a linear, procedural runner that executes a chain of responsibility (a sequence of isolated tasks or "steps") to process data safely and predictably.
+
+## 2. Philosophy (Why does it exist?)
+When data transformations are written as one large function, handling specific errors (like a network timeout on step 4 of 10) is a nightmare. The Orchestrator exists to enforce the single-responsibility principle. By breaking an ETL job down into small, functional steps that only mutate a shared `context` dictionary, developers can easily unit test individual transformations, insert retry logic automatically, and halt the pipeline gracefully upon failure.
+
+## 3. Target Audience (Who is it for?)
+This module is for Data Engineers who are writing complex ETL or ELT jobs, data migration scripts, or nightly batch processes that require clear execution boundaries, error isolation, and detailed logging at each stage.
+
+## 4. Architecture (How does it work?)
+- **Shared Context**: The orchestrator initializes a `context` (usually a dictionary) that is passed sequentially from one step to the next.
+- **Adding Steps**: Tasks are registered in the orchestrator via `add_step(name, function)`. The function must accept the `context` as an argument and return it.
+- **Execution Engine**: The `execute()` method runs the steps in order. Because it manages the execution loop, it can natively implement advanced features like automated retry policies, step-timing metrics, or alert dispatching if a specific node crashes.
+
+## 5. Installation / Setup
+The `PipelineOrchestrator` is natively included in the `ferrox-py-utils` package. It uses standard Python synchronous or asynchronous mechanics and requires no external dependencies.
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py_utils.pipelines.orchestrator import PipelineOrchestrator
+
+# 1. Define isolated, pure functions (Steps)
+def extract_data(ctx):
+    ctx['raw_data'] = [1, 2, 3]
+    return ctx
+
+def transform_data(ctx):
+    # Mutates the shared context
+    ctx['transformed'] = [x * 10 for x in ctx['raw_data']]
+    return ctx
+
+# 2. Build the Pipeline
+pipeline = PipelineOrchestrator()
+pipeline.add_step("Extraction Step", extract_data)
+pipeline.add_step("Transformation Step", transform_data)
+
+# 3. Execute the chain
+final_context = pipeline.execute()
+assert final_context['transformed'] == [10, 20, 30]
+```
+
+## 7. Ecosystem Integration
+The `PipelineOrchestrator` relies heavily on the **Observability Component** from the core `ferrox-py` framework. Because the orchestrator controls the execution loop, it automatically injects logging trace IDs and emits structured logs (e.g., `Step 'Extraction Step' completed in 0.4s`) into the centralized logging system, ensuring complete visibility over background data jobs.

ferrox_py_utils-1.0.0/docs/schemas.md

@@ -0,0 +1,47 @@
+# Schema Registry
+
+## 1. Overview (What does this do?)
+The `SchemaRegistry` module guarantees data quality during ETL processes. It provides a centralized dictionary that maps string identifiers to Pydantic models (or Dataclasses), allowing the system to perform rigorous data casting and validation on raw, untyped datasets.
+
+## 2. Philosophy (Why does it exist?)
+When ingesting data from external sources (like a third-party CSV or a public JSON API), the data is inherently untrusted and untyped. Directly inserting this raw data into the Database (the Bronze/Silver/Gold layers) leads to corrupted records and fatal crashes. This module enforces the philosophy that data must be formally validated at the application boundary. If a CSV row says an ID is `"123"`, the Schema Registry mathematically proves and casts it to an `int(123)` before it is allowed further into the pipeline.
+
+## 3. Target Audience (Who is it for?)
+This component is vital for Data Engineers who manage data lakes or data warehouses, and backend developers who need to ensure that bulk data imports adhere strictly to the internal domain models of the application.
+
+## 4. Architecture (How does it work?)
+- **Registration**: At application boot, developers map string names (e.g., `"user_import"`) to corresponding Pydantic schemas.
+- **Validation**: During a pipeline execution, a transformation step can invoke `SchemaRegistry.validate("user_import", raw_dict)`. The registry locates the schema and processes the dictionary.
+- **Error Handling**: If the raw data violates the schema (e.g., a missing required field or wrong data type), Pydantic raises a detailed validation error, allowing the orchestrator to log the specific malformed row and continue or halt appropriately.
+
+## 5. Installation / Setup
+The Schema Registry relies heavily on the `pydantic` library to perform its fast, strict type checking.
+
+```bash
+pip install pydantic
+```
+
+## 6. Quickstart (Usage)
+```python
+from ferrox_py_utils.schemas.registry import SchemaRegistry
+from pydantic import BaseModel
+
+# 1. Define the strict data contract
+class UserImportSchema(BaseModel):
+    id: int
+    email: str
+
+# 2. Register the schema centrally
+registry = SchemaRegistry()
+registry.register("user_import", UserImportSchema)
+
+# 3. Automatic casting and validation inside a Pipeline step
+raw_data = {"id": "123", "email": "test@test.com"}
+valid_user = registry.validate("user_import", raw_data)
+
+# The string "123" has been safely cast to an integer
+print(type(valid_user.id))  # <class 'int'>
+```
+
+## 7. Ecosystem Integration
+The Schema Registry serves the exact same purpose in the Data Engineering layer as the **Validation Pipe** (Layer 5) serves in the Web/HTTP layer of the core `ferrox-py` framework. It ensures that the **CQRS Bus** and **Data Repositories** only ever interact with strongly-typed, predictable Domain Objects.

ferrox_py_utils-1.0.0/ferrox_py_utils/__init__.py

@@ -0,0 +1 @@
+# init

ferrox_py_utils-1.0.0/ferrox_py_utils/connectors/__init__.py

@@ -0,0 +1 @@
+# init

ferrox_py_utils-1.0.0/ferrox_py_utils/connectors/base.py

@@ -0,0 +1,19 @@
+from abc import ABC, abstractmethod
+from typing import AsyncGenerator, Any
+
+class DataConnector(ABC):
+    @abstractmethod
+    async def connect(self):
+        pass
+
+    @abstractmethod
+    async def extract(self, query: str = None) -> AsyncGenerator[Any, None]:
+        pass
+        
+    @abstractmethod
+    async def load(self, data: Any) -> bool:
+        pass
+
+    @abstractmethod
+    async def close(self):
+        pass

ferrox_py_utils-1.0.0/ferrox_py_utils/connectors/csv.py

@@ -0,0 +1,35 @@
+import csv
+import aiofiles
+from typing import AsyncGenerator, Any
+from .base import DataConnector
+from ferrox_py.core.provider import injectable
+
+@injectable()
+class CsvConnector(DataConnector):
+    def __init__(self, file_path: str):
+        self.file_path = file_path
+        self._file = None
+
+    async def connect(self):
+        # In a real app we'd keep it open for stream
+        pass
+
+    async def extract(self, query: str = None) -> AsyncGenerator[dict, None]:
+        async with aiofiles.open(self.file_path, mode='r', encoding='utf-8') as f:
+            header = None
+            async for line in f:
+                row = line.strip().split(",")
+                if not header:
+                    header = row
+                    continue
+                yield dict(zip(header, row))
+
+    async def load(self, data: Any) -> bool:
+        # Simplistic append
+        async with aiofiles.open(self.file_path, mode='a', encoding='utf-8') as f:
+            if isinstance(data, dict):
+                await f.write(",".join(str(v) for v in data.values()) + "\n")
+        return True
+
+    async def close(self):
+        pass

ferrox_py_utils-1.0.0/ferrox_py_utils/connectors/s3.py

@@ -0,0 +1,26 @@
+from typing import AsyncGenerator, Any
+from .base import DataConnector
+from ferrox_py.core.provider import injectable
+
+@injectable()
+class S3Connector(DataConnector):
+    def __init__(self, bucket: str, path: str):
+        self.bucket = bucket
+        self.path = path
+        # Would inject aioboto3 session here
+
+    async def connect(self):
+        print(f"Connecting to S3 Bucket: {self.bucket}...")
+        pass
+
+    async def extract(self, query: str = None) -> AsyncGenerator[Any, None]:
+        print(f"Extracting streaming chunks from s3://{self.bucket}/{self.path}")
+        # Mock streaming chunks
+        yield {"chunk_id": 1, "data": b"mock_data"}
+
+    async def load(self, data: Any) -> bool:
+        print(f"Uploading chunk to s3://{self.bucket}/{self.path}")
+        return True
+
+    async def close(self):
+        pass

ferrox_py_utils-1.0.0/ferrox_py_utils/pipelines/__init__.py

@@ -0,0 +1 @@
+# init

ferrox_py_utils-1.0.0/ferrox_py_utils/pipelines/orchestrator.py

@@ -0,0 +1,28 @@
+from typing import Callable, List, Any
+import asyncio
+from ferrox_py.core.provider import injectable
+from ferrox_py.core.errors import FerroxError
+
+class PipelineStep:
+    def __init__(self, name: str, execute_fn: Callable[[Any], Any]):
+        self.name = name
+        self.execute_fn = execute_fn
+
+@injectable()
+class PipelineOrchestrator:
+    async def execute_pipeline(self, name: str, initial_data: Any, steps: List[PipelineStep]) -> Any:
+        print(f"Starting Pipeline: {name}")
+        current_data = initial_data
+        
+        for step in steps:
+            print(f"  -> Executing Step: {step.name}")
+            try:
+                if asyncio.iscoroutinefunction(step.execute_fn):
+                    current_data = await step.execute_fn(current_data)
+                else:
+                    current_data = step.execute_fn(current_data)
+            except Exception as e:
+                raise FerroxError(message=f"Pipeline '{name}' failed at step '{step.name}': {e}", status_code=500)
+                
+        print(f"Pipeline '{name}' finished successfully.")
+        return current_data

ferrox_py_utils-1.0.0/ferrox_py_utils/schemas/__init__.py

@@ -0,0 +1 @@
+# init

ferrox_py_utils-1.0.0/ferrox_py_utils/schemas/registry.py

@@ -0,0 +1,25 @@
+from typing import Dict, Type
+from pydantic import create_model, BaseModel, ValidationError
+from ferrox_py.core.provider import injectable
+from ferrox_py.core.errors import FerroxError
+
+@injectable()
+class SchemaRegistry:
+    def __init__(self):
+        self._schemas: Dict[str, Type[BaseModel]] = {}
+
+    def register_schema(self, name: str, schema_def: Dict[str, Type]):
+        model = create_model(name, **schema_def)
+        self._schemas[name] = model
+        print(f"Schema '{name}' registered.")
+
+    def validate(self, name: str, data: dict) -> dict:
+        if name not in self._schemas:
+            raise FerroxError(message=f"Schema {name} not found", status_code=404)
+            
+        model = self._schemas[name]
+        try:
+            instance = model(**data)
+            return instance.model_dump()
+        except ValidationError as e:
+            raise FerroxError(message=f"Schema validation failed: {e.errors()}", status_code=400)

ferrox_py_utils-1.0.0/pyproject.toml

@@ -0,0 +1,15 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "ferrox-py-utils"
+version = "1.0.0"
+description = "Data Engineering and ETL utilities for the Ferrox ecosystem."
+authors = [{ name = "AI-Autistic-Intelligence" }]
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "ferrox-py>=1.0.0",
+    "pydantic>=2.0"
+]