sibi-flux 2025.12.0__tar.gz

sibi_flux-2025.12.0/PKG-INFO

@@ -0,0 +1,283 @@
+Metadata-Version: 2.3
+Name: sibi-flux
+Version: 2025.12.0
+Summary: Sibi Toolkit: A collection of tools for Data Analysis/Engineering.
+Author: Luis Valverde
+Author-email: Luis Valverde <lvalverdeb@gmail.com>
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: pyarrow>=22.0.0
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: pydantic-settings>=2.12.0
+Requires-Dist: dask>=2025.11.0
+Requires-Dist: fsspec>=2025.10.0
+Requires-Dist: s3fs>=2025.10.0
+Requires-Dist: sqlalchemy>=2.0.44
+Requires-Dist: psycopg2>=2.9.11
+Requires-Dist: pymysql>=1.1.2
+Requires-Dist: clickhouse-connect>=0.10.0
+Requires-Dist: concurrent-log-handler>=0.9.28
+Requires-Dist: rich>=14.2.0
+Requires-Dist: filelock>=3.20.1
+Requires-Dist: tqdm>=4.67.1
+Requires-Dist: watchdog>=6.0.0
+Requires-Dist: tornado==6.5.4
+Requires-Dist: typer>=0.21.0
+Requires-Dist: psutil>=6.1.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: opentelemetry-api>=1.38.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.38.0
+Requires-Dist: opentelemetry-sdk>=1.38.0
+Requires-Dist: sibi-flux[distributed,geospatial,mcp] ; extra == 'complete'
+Requires-Dist: distributed>=2025.11.0 ; extra == 'distributed'
+Requires-Dist: osmnx>=2.0.7 ; extra == 'geospatial'
+Requires-Dist: geopandas>=1.1.2 ; extra == 'geospatial'
+Requires-Dist: geopy>=2.4.1 ; extra == 'geospatial'
+Requires-Dist: folium>=0.20.0 ; extra == 'geospatial'
+Requires-Dist: osmium>=4.2.0 ; extra == 'geospatial'
+Requires-Dist: shapely>=2.0.0 ; extra == 'geospatial'
+Requires-Dist: networkx>=3.6.1 ; extra == 'geospatial'
+Requires-Dist: sibi-flux[distributed] ; extra == 'mcp'
+Requires-Dist: mcp>=1.1.2 ; extra == 'mcp'
+Requires-Dist: fastapi>=0.127.0 ; extra == 'mcp'
+Requires-Dist: uvicorn>=0.40.0 ; extra == 'mcp'
+Requires-Dist: httpx>=0.28.1 ; extra == 'mcp'
+Requires-Python: >=3.11
+Provides-Extra: complete
+Provides-Extra: distributed
+Provides-Extra: geospatial
+Provides-Extra: mcp
+Description-Content-Type: text/markdown
+
+# SibiFlux
+
+**SibiFlux** is a production-grade resilient data engineering ecosystem designed to bridge the gap between local development, distributed computing, and agentic AI workflows. It provides a unified engine for hybrid data loading (batch + streaming), self-healing distributed operations, and native interfaces for AI agents via the Model Context Protocol (MCP).
+
+```mermaid
+graph TD
+    subgraph "Agentic Interface (MCP)"
+        Agent["AI Agent / Claude"] <--> Router["MCP Router"]
+        Router <--> Resources["SibiFlux Resources"]
+    end
+
+    subgraph "Solutions Layer (Business Logic)"
+        Logistics["Logistics Solutions"]
+        Enrichment["Enrichment Pipelines"]
+        Cubes["DataCubes"]
+    end
+
+    subgraph "SibiFlux Core Engine"
+        DfHelper["DfHelper (Unified Loader)"]
+        Cluster["Resilient Dask Cluster"]
+        Managed["ManagedResource Lifecycle"]
+    end
+
+    Resources --> Cubes
+    Logistics --> DfHelper
+    Cubes --> DfHelper
+    DfHelper --> Cluster
+```
+
+## Core Architecture
+
+### 1. The Flux Engine (`sibi_flux`)
+The foundational library providing resilient distributed primitives.
+- **`DfHelper`**: A unified API for loading data from SQLAlchemy, Parquet, or HTTP sources into Pandas or Dask DataFrames.
+- **`Dataset`**: A high-level abstraction for hybrid data loading, seamlessly merging historical (Parquet) and live (SQL) data sources.
+- **`DfValidator`**: A robust schema enforcement tool that validates DataFrames against strict type maps and generates ClickHouse DDL.
+- **`ArtifactOrchestrator`**: An async engine for managing concurrent artifact updates with retries, backoff, and worker isolation.
+- **`ManagedResource`**: A rigorous lifecycle management system for async resources, ensuring clean shutdown, signal handling, and observability.
+- **`Dask Cluster`**: A self-healing distributed runtime that detects worker failures, manages re-connection, and enforces "The Nuclear Option" for test isolation.
+
+### 2. Agentic Interface (`mcp`)
+Native support for the **Model Context Protocol (MCP)**, allowing AI agents to directly interact with SibiFlux data structures.
+- **Expose DataCubes**: Automatically turn any `DataCube` into a queryable MCP Resource.
+- **Tooling**: Register Python functions as tools callable by agents.
+
+## Key Capabilities
+
+### Hybrid Data Loading
+SibiFlux implements a "Hot/Cold" architecture for seamless data access:
+- **Historical Data**: Read efficiently from partitioned Parquet archives (S3/Local).
+- **Live Data**: Query operational SQL databases for real-time changes.
+- **Automatic Merge**: `DfHelper` and `Dataset` automatically stitch these sources together, handling schema evolution and deduplication.
+
+### Resilient Distributed Compute
+The SibiFlux Dask wrapper provides:
+- **Auto-Healing**: Clients that automatically reconnect if the scheduler dies.
+- **Safe Persistence**: Wrappers like `safe_persist` that retry operations on network jitter.
+- **Smart Partitioning**: Automated repartitioning to prevent "small file problem" in Parquet outputs.
+
+### Observability
+Built-in integration with OpenTelemetry (OTel):
+- structured logging with correlation IDs.
+- distributed tracing across async boundaries.
+
+## Quick Start
+
+### Installation
+
+```bash
+# Base installation (Core Engine only)
+pip install sibi-flux
+
+# For Distributed Computing (Dask Cluster support)
+pip install "sibi-flux[distributed]"
+
+# For Geospatial capabilities (OSMnx, GeoPandas, etc.)
+pip install "sibi-flux[geospatial]"
+
+# For MCP Agentic Interface support
+pip install "sibi-flux[mcp]"
+
+# Complete installation (Simulates the "All-in-One" environment)
+pip install "sibi-flux[complete]"
+```
+
+## API Examples
+
+### 1. Data Loading (`Dataset`)
+
+The `Dataset` class provides a high-level abstraction for hybrid loading.
+
+```python
+from sibi_flux import Dataset
+from solutions.logistics.readers.products import ProductsParquetReader, ProductsSqlReader
+
+class ProductsDataset(Dataset):
+    historical_reader = ProductsParquetReader
+    live_reader = ProductsSqlReader
+    date_field = "created_at"
+
+# Load hybrid data (merges Parquet + SQL)
+ds = ProductsDataset(start_date="2023-01-01", end_date="2023-01-31")
+df = await ds.aload()  # Returns a Dask DataFrame
+```
+
+### 2. Schema Validation (`DfValidator`)
+
+Ensure your data meets strict type requirements and generate DDL for ClickHouse.
+
+```python
+from sibi_flux import DfValidator
+
+# Define expected schema
+SCHEMA = {
+    "product_id": "Int64[pyarrow]",
+    "price": "Float64[pyarrow]",
+    "name": "string[pyarrow]"
+}
+
+validator = DfValidator(df)
+validator.validate_schema(SCHEMA)
+validator.standardize_data_quality()
+
+# Generate ClickHouse DDL
+ddl = validator.generate_clickhouse_ddl("products_table")
+print(ddl)
+```
+
+### 3. Data Enrichment (`AsyncDfEnricher`)
+
+Enrich a base DataFrame with data from other sources using `AttachmentSpec`.
+
+```python
+from sibi_flux.df_enricher import AsyncDfEnricher, AttachmentSpec
+
+specs = [
+    AttachmentSpec(
+        key="customer_info",
+        required_cols={"customer_id"},
+        attachment_fn=fetch_customer_data,  # Async function returning DF
+        left_on=["customer_id"],
+        right_on=["id"],
+        drop_cols=["id"]
+    )
+]
+
+enricher = AsyncDfEnricher(base_df=orders_df, specs=specs)
+enriched_df = await enricher.enrich()
+```
+
+### 4. Orchestration (`ArtifactOrchestrator`)
+
+Manage concurrent updates of multiple artifacts with retries and worker isolation.
+
+```python
+from sibi_flux.orchestration import ArtifactOrchestrator
+
+orchestrator = ArtifactOrchestrator(
+    wrapped_classes={"daily": [ProductsDataset, OrdersDataset]},
+    max_workers=3,
+    retry_attempts=3
+)
+
+# Updates all 'daily' artifacts concurrently
+results = await orchestrator.update_data("daily")
+```
+
+### 5. Distributed Compute (`Dask Cluster`)
+
+Execute resilient operations that survive scheduler restarts.
+
+```python
+from sibi_flux.dask_cluster import safe_compute, get_persistent_client
+
+client = get_persistent_client()
+
+# Safe compute with auto-retry logic
+result = safe_compute(df.groupby("category").sum())
+```
+
+### 6. Resource Management (`ManagedResource`)
+
+Create lifecycle-safe components.
+
+```python
+from sibi_flux.core import ManagedResource
+
+class MyResource(ManagedResource):
+    async def _acleanup(self):
+        await self.db_connection.close()
+        self.logger.info("Cleaned up!")
+
+async with MyResource() as res:
+    await res.do_work()
+# Automatically cleaned up here
+```
+
+### 7. Agentic Interface (`sibi_flux.mcp`)
+
+Seamlessly bridge your data with AI Agents.
+
+**Server Side (Expose Resources)**
+
+```python
+from sibi_flux.mcp import BaseMCPRouter
+from products import ProductsDataset
+
+# Create an MCP Router compatible with FastAPI
+router = BaseMCPRouter(name="data-server")
+
+# Automatically register a Dataset as an MCP Resource
+# Agent can now read `sibi://ProductsDataset`
+router.register_cube_resource(ProductsDataset)
+
+# Register a custom tool
+@router.tool()
+def calculate_vat(amount: float) -> float:
+    return amount * 0.2
+```
+
+**Client Side (Consume Resources)**
+
+```python
+from sibi_flux.mcp import GenericMcpClient
+
+# Connect to the MCP Server
+async with GenericMcpClient(url="http://localhost:8000/sse") as client:
+    # Read the resource (returns JSON data from the Dataset)
+    data = await client.read_resource("sibi://ProductsDataset")
+    
+    # Call a tool
+    vat = await client.call_tool("calculate_vat", arguments={"amount": 100.0})
+```

sibi_flux-2025.12.0/README.md

@@ -0,0 +1,233 @@
+# SibiFlux
+
+**SibiFlux** is a production-grade resilient data engineering ecosystem designed to bridge the gap between local development, distributed computing, and agentic AI workflows. It provides a unified engine for hybrid data loading (batch + streaming), self-healing distributed operations, and native interfaces for AI agents via the Model Context Protocol (MCP).
+
+```mermaid
+graph TD
+    subgraph "Agentic Interface (MCP)"
+        Agent["AI Agent / Claude"] <--> Router["MCP Router"]
+        Router <--> Resources["SibiFlux Resources"]
+    end
+
+    subgraph "Solutions Layer (Business Logic)"
+        Logistics["Logistics Solutions"]
+        Enrichment["Enrichment Pipelines"]
+        Cubes["DataCubes"]
+    end
+
+    subgraph "SibiFlux Core Engine"
+        DfHelper["DfHelper (Unified Loader)"]
+        Cluster["Resilient Dask Cluster"]
+        Managed["ManagedResource Lifecycle"]
+    end
+
+    Resources --> Cubes
+    Logistics --> DfHelper
+    Cubes --> DfHelper
+    DfHelper --> Cluster
+```
+
+## Core Architecture
+
+### 1. The Flux Engine (`sibi_flux`)
+The foundational library providing resilient distributed primitives.
+- **`DfHelper`**: A unified API for loading data from SQLAlchemy, Parquet, or HTTP sources into Pandas or Dask DataFrames.
+- **`Dataset`**: A high-level abstraction for hybrid data loading, seamlessly merging historical (Parquet) and live (SQL) data sources.
+- **`DfValidator`**: A robust schema enforcement tool that validates DataFrames against strict type maps and generates ClickHouse DDL.
+- **`ArtifactOrchestrator`**: An async engine for managing concurrent artifact updates with retries, backoff, and worker isolation.
+- **`ManagedResource`**: A rigorous lifecycle management system for async resources, ensuring clean shutdown, signal handling, and observability.
+- **`Dask Cluster`**: A self-healing distributed runtime that detects worker failures, manages re-connection, and enforces "The Nuclear Option" for test isolation.
+
+### 2. Agentic Interface (`mcp`)
+Native support for the **Model Context Protocol (MCP)**, allowing AI agents to directly interact with SibiFlux data structures.
+- **Expose DataCubes**: Automatically turn any `DataCube` into a queryable MCP Resource.
+- **Tooling**: Register Python functions as tools callable by agents.
+
+## Key Capabilities
+
+### Hybrid Data Loading
+SibiFlux implements a "Hot/Cold" architecture for seamless data access:
+- **Historical Data**: Read efficiently from partitioned Parquet archives (S3/Local).
+- **Live Data**: Query operational SQL databases for real-time changes.
+- **Automatic Merge**: `DfHelper` and `Dataset` automatically stitch these sources together, handling schema evolution and deduplication.
+
+### Resilient Distributed Compute
+The SibiFlux Dask wrapper provides:
+- **Auto-Healing**: Clients that automatically reconnect if the scheduler dies.
+- **Safe Persistence**: Wrappers like `safe_persist` that retry operations on network jitter.
+- **Smart Partitioning**: Automated repartitioning to prevent "small file problem" in Parquet outputs.
+
+### Observability
+Built-in integration with OpenTelemetry (OTel):
+- structured logging with correlation IDs.
+- distributed tracing across async boundaries.
+
+## Quick Start
+
+### Installation
+
+```bash
+# Base installation (Core Engine only)
+pip install sibi-flux
+
+# For Distributed Computing (Dask Cluster support)
+pip install "sibi-flux[distributed]"
+
+# For Geospatial capabilities (OSMnx, GeoPandas, etc.)
+pip install "sibi-flux[geospatial]"
+
+# For MCP Agentic Interface support
+pip install "sibi-flux[mcp]"
+
+# Complete installation (Simulates the "All-in-One" environment)
+pip install "sibi-flux[complete]"
+```
+
+## API Examples
+
+### 1. Data Loading (`Dataset`)
+
+The `Dataset` class provides a high-level abstraction for hybrid loading.
+
+```python
+from sibi_flux import Dataset
+from solutions.logistics.readers.products import ProductsParquetReader, ProductsSqlReader
+
+class ProductsDataset(Dataset):
+    historical_reader = ProductsParquetReader
+    live_reader = ProductsSqlReader
+    date_field = "created_at"
+
+# Load hybrid data (merges Parquet + SQL)
+ds = ProductsDataset(start_date="2023-01-01", end_date="2023-01-31")
+df = await ds.aload()  # Returns a Dask DataFrame
+```
+
+### 2. Schema Validation (`DfValidator`)
+
+Ensure your data meets strict type requirements and generate DDL for ClickHouse.
+
+```python
+from sibi_flux import DfValidator
+
+# Define expected schema
+SCHEMA = {
+    "product_id": "Int64[pyarrow]",
+    "price": "Float64[pyarrow]",
+    "name": "string[pyarrow]"
+}
+
+validator = DfValidator(df)
+validator.validate_schema(SCHEMA)
+validator.standardize_data_quality()
+
+# Generate ClickHouse DDL
+ddl = validator.generate_clickhouse_ddl("products_table")
+print(ddl)
+```
+
+### 3. Data Enrichment (`AsyncDfEnricher`)
+
+Enrich a base DataFrame with data from other sources using `AttachmentSpec`.
+
+```python
+from sibi_flux.df_enricher import AsyncDfEnricher, AttachmentSpec
+
+specs = [
+    AttachmentSpec(
+        key="customer_info",
+        required_cols={"customer_id"},
+        attachment_fn=fetch_customer_data,  # Async function returning DF
+        left_on=["customer_id"],
+        right_on=["id"],
+        drop_cols=["id"]
+    )
+]
+
+enricher = AsyncDfEnricher(base_df=orders_df, specs=specs)
+enriched_df = await enricher.enrich()
+```
+
+### 4. Orchestration (`ArtifactOrchestrator`)
+
+Manage concurrent updates of multiple artifacts with retries and worker isolation.
+
+```python
+from sibi_flux.orchestration import ArtifactOrchestrator
+
+orchestrator = ArtifactOrchestrator(
+    wrapped_classes={"daily": [ProductsDataset, OrdersDataset]},
+    max_workers=3,
+    retry_attempts=3
+)
+
+# Updates all 'daily' artifacts concurrently
+results = await orchestrator.update_data("daily")
+```
+
+### 5. Distributed Compute (`Dask Cluster`)
+
+Execute resilient operations that survive scheduler restarts.
+
+```python
+from sibi_flux.dask_cluster import safe_compute, get_persistent_client
+
+client = get_persistent_client()
+
+# Safe compute with auto-retry logic
+result = safe_compute(df.groupby("category").sum())
+```
+
+### 6. Resource Management (`ManagedResource`)
+
+Create lifecycle-safe components.
+
+```python
+from sibi_flux.core import ManagedResource
+
+class MyResource(ManagedResource):
+    async def _acleanup(self):
+        await self.db_connection.close()
+        self.logger.info("Cleaned up!")
+
+async with MyResource() as res:
+    await res.do_work()
+# Automatically cleaned up here
+```
+
+### 7. Agentic Interface (`sibi_flux.mcp`)
+
+Seamlessly bridge your data with AI Agents.
+
+**Server Side (Expose Resources)**
+
+```python
+from sibi_flux.mcp import BaseMCPRouter
+from products import ProductsDataset
+
+# Create an MCP Router compatible with FastAPI
+router = BaseMCPRouter(name="data-server")
+
+# Automatically register a Dataset as an MCP Resource
+# Agent can now read `sibi://ProductsDataset`
+router.register_cube_resource(ProductsDataset)
+
+# Register a custom tool
+@router.tool()
+def calculate_vat(amount: float) -> float:
+    return amount * 0.2
+```
+
+**Client Side (Consume Resources)**
+
+```python
+from sibi_flux.mcp import GenericMcpClient
+
+# Connect to the MCP Server
+async with GenericMcpClient(url="http://localhost:8000/sse") as client:
+    # Read the resource (returns JSON data from the Dataset)
+    data = await client.read_resource("sibi://ProductsDataset")
+    
+    # Call a tool
+    vat = await client.call_tool("calculate_vat", arguments={"amount": 100.0})
+```

sibi_flux-2025.12.0/pyproject.toml

@@ -0,0 +1,163 @@
+[project]
+name = "sibi-flux"
+version = "2025.12.0"
+description = "Sibi Toolkit: A collection of tools for Data Analysis/Engineering."
+readme = "README.md"
+authors = [
+    { name = "Luis Valverde", email = "lvalverdeb@gmail.com" }
+]
+requires-python = ">=3.11"
+dependencies = [
+    # Core Dependencies
+    "pandas>=2.3.3",
+    "pyarrow>=22.0.0",
+    "pydantic>=2.12.5",
+    "pydantic-settings>=2.12.0",
+    "dask>=2025.11.0",
+    "fsspec>=2025.10.0",
+    "s3fs>=2025.10.0",
+    "sqlalchemy>=2.0.44",
+    "psycopg2>=2.9.11",
+    "pymysql>=1.1.2",
+    "clickhouse-connect>=0.10.0",
+    "concurrent-log-handler>=0.9.28",
+    "rich>=14.2.0",
+    "filelock>=3.20.1",
+    "tqdm>=4.67.1",
+    "watchdog>=6.0.0", # Core utils usage
+    "tornado==6.5.4",
+    "typer>=0.21.0",
+    "psutil>=6.1.1",
+    "httpx>=0.28.1",
+    # Logger Core Dependencies
+    "opentelemetry-api>=1.38.0",
+    "opentelemetry-exporter-otlp>=1.38.0",
+    "opentelemetry-sdk>=1.38.0",
+]
+
+[project.optional-dependencies]
+distributed = [
+    "distributed>=2025.11.0",
+]
+mcp = [
+    "sibi-flux[distributed]",
+    "mcp>=1.1.2",
+    "fastapi>=0.127.0",
+    "uvicorn>=0.40.0",
+    "httpx>=0.28.1",
+]
+geospatial = [
+    "osmnx>=2.0.7",
+    "geopandas>=1.1.2",
+    "geopy>=2.4.1",
+    "folium>=0.20.0",
+    "osmium>=4.2.0",
+    "shapely>=2.0.0",
+    "networkx>=3.6.1",
+]
+complete = [
+    "sibi-flux[distributed,geospatial,mcp]"
+]
+
+
+[dependency-groups]
+dev = [
+    "black>=25.11.0",
+    "bokeh>=3.8.0",
+    "graphviz>=0.21",
+    "jupyter>=1.1.1",
+    "poethepoet>=0.38.0",
+    "notebook>=7.5.0",
+    "pytest>=9.0.1",
+    "pytest-asyncio>=1.3.0",
+    "ruff>=0.14.9",
+    "httpx>=0.28.1",
+]
+geospatial = [
+    "folium>=0.20.0",
+    "geopandas>=1.1.2",
+    "geopy>=2.4.1",
+    "networkx>=3.6.1",
+    "osmnx>=2.0.7",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.5,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = "src"
+module-name = ["sibi_flux", "sibi_dst"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src", "."]
+testpaths = ["tests"]
+addopts = "-v"
+filterwarnings = ["ignore::DeprecationWarning"]
+
+
+[tool.poe.tasks]
+dev = """
+uvicorn solutions.main:app
+    --reload
+    --reload-dir solutions
+    --host 0.0.0.0
+    --port 6500    
+    --env-file .env.linux
+    --workers 1
+"""
+test = { cmd = "pytest tests/"}
+lint = "black src/"
+
+[tool.commitizen]
+name = "cz_conventional_commits"
+version = "1.0.0"
+tag_format = "v$version"
+
+[tool.mypy]
+python_version = "3.11"
+mypy_path = "src"
+explicit_package_bases = true
+namespace_packages = true
+warn_return_any = false
+warn_unused_configs = true
+check_untyped_defs = true
+disallow_untyped_defs = false 
+ignore_missing_imports = false
+
+# Ignore missing imports for specific 3rd party libraries without strict typing
+[[tool.mypy.overrides]]
+module = [
+    "pandas.*",
+    "dask.*",
+    "distributed.*",
+    "fsspec.*",
+    "s3fs.*",
+    "sqlalchemy.*",
+    "pyarrow.*",
+    "clickhouse_connect.*",
+    "rich.*",
+    "filelock.*",
+    "tqdm.*",
+    "watchdog.*",
+    "tornado.*",
+    "typer.*",
+    "fastapi.*",
+    "uvicorn.*",
+    "httpx.*",
+    "osmnx.*",
+    "geopandas.*",
+    "geopy.*",
+    "folium.*",
+    "osmium.*",
+    "shapely.*",
+    "networkx.*",
+    "opentelemetry.*",
+    "psutil.*",
+    "pytest.*",
+    "mcp.*",
+    "yaml.*",
+    "pydantic.*",
+    "pydantic_settings.*",
+]
+ignore_missing_imports = true