zoopipe 2026.1.20__cp310-abi3-macosx_11_0_arm64.whl

zoopipe-2026.1.20.dist-info/METADATA

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: zoopipe
+Version: 2026.1.20
+Requires-Dist: pydantic>=2.0
+Requires-Dist: dask[distributed]>=2026.1.1 ; extra == 'dask'
+Requires-Dist: mkdocs>=1.6.1 ; extra == 'docs'
+Requires-Dist: mkdocs-material>=9.7.1 ; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=1.0.0 ; extra == 'docs'
+Requires-Dist: ray>=2.53.0 ; extra == 'ray'
+Provides-Extra: dask
+Provides-Extra: docs
+Provides-Extra: ray
+License-File: LICENSE
+Summary: ZooPipe is a data processing framework that allows you to process data in a declarative way.
+Author-email: Alberto Daniel Badia <alberto_badia@enlacepatagonia.com>
+Requires-Python: >=3.10, <3.14
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Homepage, https://github.com/albertobadia/zoopipe
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/logo-dark.svg">
+    <source media="(prefers-color-scheme: light)" srcset="docs/assets/logo-light.svg">
+    <img alt="ZooPipe Logo" src="docs/assets/logo-light.svg" width="600">
+  </picture>
+</p>
+
+**ZooPipe** is a lean, ultra-high-performance data processing engine for Python. It leverages a **100% Rust core** to handle I/O and orchestration, while keeping the flexibility of Python for schema validation (via Pydantic) and custom data enrichment (via Hooks).
+
+<p align="center">
+  <a href="https://pypi.org/project/zoopipe/"><img alt="PyPI" src="https://img.shields.io/pypi/v/zoopipe"></a>
+  <img alt="Downloads" src="https://img.shields.io/pypi/dm/zoopipe">
+  <a href="https://github.com/albertobadia/zoopipe/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/albertobadia/zoopipe/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://zoopipe.readthedocs.io/"><img alt="ReadTheDocs" src="https://img.shields.io/readthedocs/zoopipe"></a>
+</p>
+
+---
+
+Read the [docs](https://zoopipe.readthedocs.io/) for more information.
+
+## ✨ Key Features
+
+- 🚀 **100% Native Rust Engine**: The core execution loop, including CSV and JSON parsing/writing, is implemented in Rust for maximum throughput.
+- 🔍 **Declarative Validation**: Use [Pydantic](https://docs.pydantic.dev/) models to define and validate your data structures naturally.
+- 🪝 **Python Hooks**: Transform and enrich data at any stage using standard Python functions or classes.
+- 🚨 **Automated Error Routing**: Native support for routing failed records to a dedicated error output.
+- 📊 **Multiple Format Support**: Optimized readers/writers for CSV, JSONL, and SQL databases.
+- 🔧 **Two-Tier Parallelism**: Orchestrate across processes or clusters with **Engines** (Local, Ray), and scale throughput at the node level with Rust **Executors**.
+- ☁️ **Cloud Native**: Native S3 support and zero-config distributed execution on **Ray** clusters.
+
+---
+
+## ⚡ Performance & Benchmarks
+
+Why ZooPipe? Because **vectorization isn't always the answer.**
+
+Tools like **Pandas** and **Polars** are incredible for analytical workloads (groupby, sum, joins) where operations can be vectorized in C/Rust. However, real-world Data Engineering often involves "chaotic ETL": messy custom rules, API calls per row, hashing, conditional cleanup, and complex normalization that forcedly drop down to Python loops.
+
+**In these "Heavy ETL" scenarios, ZooPipe outperforms Vectorized DataFrames by 3x-8x.**
+
+![Benchmark Chart](docs/assets/benchmark.svg)
+
+> **Key Takeaway**: ZooPipe's "Python-First Architecture" with parallel streaming (`PipeManager`) avoids the serialization overhead that cripples Polars/Pandas when using Python UDFs (`map_elements`/`apply`), and uses **97% less RAM**.
+
+### ⚖️ Is this unfair to Pandas/Polars?
+
+**Yes and No.**
+
+- **Unfair**: If your workload is purely analytical (e.g., `GROUP BY`, `SUM`, `JOIN`), **Polars and Pandas will likely destroy ZooPipe** because they can use vectorized C/Rust operations on whole columns at once.
+- **Fair**: In real-world Data Engineering, many pipelines are "chaotic". They require custom hashing, API calls per row, conditional normalization, or complex Pydantic validation. **In these "Python-UDF heavy" scenarios, vectorization breaks down**, and ZooPipe shines by orchestrating parallel Python execution efficiently without the DataFrame overhead.
+
+### ❓ When to use what?
+
+| Use **ZooPipe** When... | Use **Pandas / Polars** When... |
+|---|---|
+| 🏗️ You have complex, custom Python logic per row (hash, clean, validate). | 🧮 You are doing aggregations (SUM, AVG) or Relational Algebra (JOIN, GROUP BY). |
+| 🔄 You are processing streaming data or files larger than RAM. | 💾 Your dataset fits comfortably in RAM (or use LazyFrames). |
+| 🛡️ You need strict schema validation (Pydantic) and error handling. | 🔬 You are doing data exploration or statistical analysis. |
+| 🚀 You want to mix Rust I/O performance with Python flexibility. | ⚡ Your entire pipeline can be expressed in vectorized expressions. |
+
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install zoopipe
+```
+Or using uv:
+```bash
+uv add zoopipe
+```
+Or from source (uv recommended):
+```bash
+uv build
+uv run maturin develop --release
+```
+
+### Simple Example
+
+```python
+from pydantic import BaseModel, ConfigDict
+from zoopipe import CSVInputAdapter, CSVOutputAdapter, Pipe
+
+
+class UserSchema(BaseModel):
+    model_config = ConfigDict(extra="ignore")
+    user_id: str
+    username: str
+    email: str
+
+
+pipe = Pipe(
+    input_adapter=CSVInputAdapter("users.csv"),
+    output_adapter=CSVOutputAdapter("processed_users.csv"),
+    error_output_adapter=CSVOutputAdapter("errors.csv"),
+    schema_model=UserSchema,
+)
+
+pipe.start()
+pipe.wait()
+
+
+print(f"Finished! Processed {pipe.report.total_processed} items.")
+```
+
+Automatically split large files or manage multiple independent workflows:
+
+```python
+from zoopipe import PipeManager, MultiProcessEngine
+
+# Create your pipe as usual (Pipe is purely declarative)
+pipe = Pipe(...)
+
+# Automatically parallelize across 4 workers
+# MultiProcessEngine() for local, RayEngine() for clusters
+manager = PipeManager.parallelize_pipe(
+    pipe, 
+    workers=4, 
+    engine=MultiProcessEngine() 
+)
+manager.start()
+manager.wait()
+```
+
+---
+
+## 📚 Documentation
+
+### Core Concepts
+
+
+#### Hooks
+
+Hooks are Python classes that allow you to intercept, transform, and enrich data at different stages of the pipeline.
+
+**[📘 Read the full Hooks Guide](docs/hooks.md)** to learn about lifecycle methods (`setup`, `execute`, `teardown`), state management, and advanced patterns like cursor pagination.
+
+### Quick Example
+
+```python
+from zoopipe import BaseHook
+
+class MyHook(BaseHook):
+    def execute(self, entries, store):
+        for entry in entries:
+            entry["raw_data"]["checked"] = True
+        return entries
+```
+
+> [!IMPORTANT]
+> If you are using a `schema_model`, the pipeline will output the contents of `validated_data` for successful records.
+> - To modify data **before** validation, use `pre_validation_hooks` and modify `entry["raw_data"]`.
+> - To modify data **after** validation (and ensure it reaches the output), use `post_validation_hooks` and modify `entry["validated_data"]`.
+
+#### Executors
+
+Executors control how ZooPipe scales **up** within a single node using Rust-managed threads. They are the engine under the hood that drives high throughput.
+
+**[📘 Read the full Executors Guide](docs/executors.md)** to understand the difference between `SingleThreadExecutor` (debug/ordered) and `MultiThreadExecutor` (high-throughput).
+
+### Input/Output Adapters
+
+#### File Formats
+
+- [**CSV Adapters**](docs/csv.md) - High-performance CSV reading and writing
+- [**JSON Adapters**](docs/json.md) - JSONL and JSON array format support
+- [**Excel Adapters**](docs/excel.md) - Read and write Excel (.xlsx) files
+- [**Parquet Adapters**](docs/parquet.md) - Columnar storage for analytics and data lakes
+- [**Arrow Adapters**](docs/arrow.md) - Apache Arrow IPC format for high-throughput interoperability
+
+#### Databases
+
+- [**SQL Adapters**](docs/sql.md) - Read from and write to SQL databases with batch optimization
+- [**SQL Pagination**](docs/sql.md#sqlpaginationinputadapter) - High-performance cursor-style pagination for large tables
+- [**DuckDB Adapters**](docs/duckdb.md) - Analytical database for OLAP workloads
+
+#### Messaging Systems
+
+- [**Kafka Adapters**](docs/kafka.md) - High-throughput messaging
+
+#### Advanced
+
+- [**Python Generator Adapters**](docs/pygen.md) - In-memory streaming and testing
+- [**Cloud Storage (S3)**](docs/cloud-storage.md) - Read and write data from Amazon S3 and compatible services
+- [**PipeManager**](docs/pipemanager.md) - Run multiple pipes in parallel for distributed processing
+- [**Ray Guide**](docs/ray.md) - Zero-config distributed execution on Ray clusters
+
+---
+
+## 🛠 Architecture
+
+ZooPipe is designed as a thin Python wrapper around a powerful Rust core, featuring a two-tier parallel architecture:
+
+1. **Orchestration Tier (Python Engines)**: 
+   - Manage distribution across processes or nodes (e.g., `MultiProcessEngine`).
+   - Handles data sharding, process lifecycle, and metrics aggregation.
+2. **Execution Tier (Rust BatchExecutors)**: 
+   - **Internal Throughput**: High-speed processing within a single process.
+   - **Adapters**: Native CSV/JSON/SQL Readers and Writers.
+   - **NativePipe**: Orchestrates the loop, fetching chunks and routing result batches.
+   - **Executors**: Multi-threaded Rust strategies to bypass the GIL within a node.
+
+---
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+

zoopipe-2026.1.20.dist-info/RECORD

@@ -0,0 +1,43 @@
+zoopipe/__init__.py,sha256=gPq396dNUZVk1-bx23FvllBRHhywPxTHpjGnnLSWOlE,2487
+zoopipe/engines/__init__.py,sha256=nOGzFAnjCxhJwIRENfenKzR75EsWB9iVSYI5DfYedfQ,145
+zoopipe/engines/base.py,sha256=7ZRrlm_84OSKNapkLYlkdRxdRwKhQIDB-jCGRNhvbo0,1147
+zoopipe/engines/dask.py,sha256=jIuGeV_6KSoQkZvNirPLbcwgMHdvSXIu9cFGrbYZTrY,7327
+zoopipe/engines/local.py,sha256=6sp-Rn6MrvLo1pHRK2KERkrMHXPpsaXMaNtQcLALEeQ,6923
+zoopipe/engines/ray.py,sha256=5EgYRlY5nzSAkVXUt0q6zkfOCOO6qb9M0HaTcjy3UFk,8907
+zoopipe/hooks/__init__.py,sha256=H6nLNAZVIPULIGpPbCfQ74sjF4kexj-f0BI5cVbkVOI,185
+zoopipe/hooks/base.py,sha256=YgXazJ0jbGe971XMet6WEFkEfezPBjK4pxqjse6nzTw,1854
+zoopipe/hooks/sql.py,sha256=VS7r2xFe90dif2Tvhhw0VFMU7guUnlnZk81CA48wpOU,2868
+zoopipe/input_adapter/__init__.py,sha256=Ravsb2U4SASvOMcLpkYZeW-Iq6ITp5xzLm_ryULXIew,908
+zoopipe/input_adapter/arrow.py,sha256=Fovs9XjMk2Qw5FPGBJm3En57ctnEjTS2vIJiO15YGKo,950
+zoopipe/input_adapter/base.py,sha256=Mbhb4oKqK1mXXbcFCeD0cf8RvLJBXRxQtHCD-2dbWg8,1440
+zoopipe/input_adapter/csv.py,sha256=0HHVfFfpL_K_xle4LS6DZuXgD2JJDcq_SeA40P2tAiQ,4882
+zoopipe/input_adapter/duckdb.py,sha256=_wTjsDvrw8yTL7OTbA9ejXVbh2CH5ZhPGNo7ZayG-Jk,1634
+zoopipe/input_adapter/excel.py,sha256=ZHkl9dDguYzdu-ExF1kM0lUScoJnwoBtFWAOZTdqSYU,1635
+zoopipe/input_adapter/json.py,sha256=ZF_aa6oFSBd0WzkYpVAGgu3kAyx9jFPZufie2uGCTng,2140
+zoopipe/input_adapter/kafka.py,sha256=AnarmVzrkgWhAgerKKzQUgNE3_62YiQgpLJ-hGe2Al0,1052
+zoopipe/input_adapter/parquet.py,sha256=56zyTT8H7V-yablPkTCU-7WlllVR4o6xPmnS1_Ca-08,2600
+zoopipe/input_adapter/pygen.py,sha256=Jfy9uefNSiKZYgY0gH6nyeMYbA_NRZ9mMBLNcx9iXjc,996
+zoopipe/input_adapter/sql.py,sha256=9lN06AgWwA94BkGopKuwu2y9QL-kxNZlJ7H3_f8Q4qw,3243
+zoopipe/manager.py,sha256=XN7EVzkr9yA_XRXDO2uCg35C_IkvP-A7CXL5Xj7XxVk,7195
+zoopipe/output_adapter/__init__.py,sha256=eX09URcg0wOQtftq8BTWQodCr690SyyJKP59zKRynYA,878
+zoopipe/output_adapter/arrow.py,sha256=6WaB7zPIFO_QxyCayn3bQqKVHqD6PPT9wp2jt-wZuCY,1449
+zoopipe/output_adapter/base.py,sha256=Zq172oMzI7Jhvz9E3t0bB0DNWv1ECU47Q2MFih3-BgE,1206
+zoopipe/output_adapter/csv.py,sha256=3l1a121ttOK6IlGAeBS0l55m0BNjmDjzX5DLCfjREoI,2136
+zoopipe/output_adapter/duckdb.py,sha256=l78gdrqSG1au5SC0hp0QYoxTICOra6TXPYs5MGGfmIM,1303
+zoopipe/output_adapter/excel.py,sha256=DU2lEHHpiuDYT9pgs1jbPCtWxq6tN5t0e73-pGacXZM,1186
+zoopipe/output_adapter/json.py,sha256=OTJv-rogiIm_5j3pozl7LaZSOt3rCON4pAx1Ab9d11U,1984
+zoopipe/output_adapter/kafka.py,sha256=dz128E17hDsgRC8oOemitsQ8HPAkaqnhj9zXb9EYlmk,993
+zoopipe/output_adapter/parquet.py,sha256=ygY65JlAswEuLqs84k9SoyZPBHvRHHao0lO7f5yAg5o,1350
+zoopipe/output_adapter/pygen.py,sha256=_e3ps55U7IzKR67JsPuNyhpWVBkDgxenKLkDI6lz5FM,811
+zoopipe/output_adapter/sql.py,sha256=MDDoWnlkGtrlGm3wJZ1U-yOdgQNJXuuFvg6Ki9-7pk4,1135
+zoopipe/pipe.py,sha256=s86DaxAByNlzygond1BxhIRDruNGutJJK0viORdewDk,9740
+zoopipe/protocols.py,sha256=kK6dhHLhUchUaGjfzuJYsDCGBxoQhZSCBxIP69yVlH4,967
+zoopipe/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+zoopipe/report.py,sha256=-fvwDgWNrtwzdynmwCouxvrUEr_M6i10Nip8fiztKkk,5250
+zoopipe/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+zoopipe/utils/dependency.py,sha256=Ms2V-hR8jtYcbe3Fgnc2qfy31XJw3aIZprYz2j878Zg,2051
+zoopipe/zoopipe_rust_core.abi3.so,sha256=3HX8XeDN2xBXfiAPEfLprJjhSTwv_sdo5pzWK-Zr6Kk,65972832
+zoopipe-2026.1.20.dist-info/METADATA,sha256=BiaX3FUmpbppYLRn-u7ilKbEBGGeP13ql5BxGsEwMaQ,9591
+zoopipe-2026.1.20.dist-info/WHEEL,sha256=vZ12AMAE5CVtd8oYbYGrz3omfHuIZCNO_3P50V00s00,104
+zoopipe-2026.1.20.dist-info/licenses/LICENSE,sha256=4WRhonN0HErkcdxwCRoaBxdR4suhdVxZj_14XXMVgtw,1077
+zoopipe-2026.1.20.dist-info/RECORD,,

zoopipe-2026.1.20.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: maturin (1.11.5)
+Root-Is-Purelib: false
+Tag: cp310-abi3-macosx_11_0_arm64

zoopipe-2026.1.20.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alberto Daniel Badia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.