dataloom-engine 0.3.0__tar.gz → 0.4.0__tar.gz

dataloom_engine-0.4.0/PKG-INFO

@@ -0,0 +1,214 @@
+Metadata-Version: 2.4
+Name: dataloom-engine
+Version: 0.4.0
+Summary: DataLoom: Um orquestrador de threads leve e eficiente para dados.
+Author-email: Dioni Padilha <dionipdl@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/dionipadilha/dataloom
+Project-URL: Changelog, https://github.com/dionipadilha/dataloom/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/dionipadilha/dataloom/issues
+Keywords: pipeline,data,threading,loom,concurrency
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: numpy
+Requires-Dist: numpy>=1.21.0; extra == "numpy"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: numpy>=1.21.0; extra == "dev"
+Dynamic: license-file
+
+# 🧵 DataLoom
+
+> A lightweight, efficient and safe multi-threaded orchestration engine for Python.
+
+🇧🇷 [Versão em português](README.pt-BR.md)
+
+[![CI](https://github.com/dionipadilha/dataloom/actions/workflows/ci.yml/badge.svg)](https://github.com/dionipadilha/dataloom/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/dataloom-engine)](https://pypi.org/project/dataloom-engine/) [![Python](https://img.shields.io/pypi/pyversions/dataloom-engine)](https://pypi.org/project/dataloom-engine/) ![License](https://img.shields.io/badge/license-MIT-green)
+
+<img width="2752" height="1536" alt="DataLoom: a lightweight, efficient and safe multi-threaded orchestration engine for Python." src="https://github.com/user-attachments/assets/c1f3fc98-5eaf-4add-a38c-2717ae699cd5" />
+
+**DataLoom** is a library designed to process data streams using the **Producer-Consumer** pattern with multiple threads (Weavers). It abstracts away the complexity of queues, locks and lifecycle management, letting you focus purely on your data transformation logic.
+
+## Why DataLoom?
+⚡ Performance: real parallel processing for I/O-bound workloads.
+🧩 Simplicity: an intuitive API inspired by the weaving metaphor.
+🛡️ Safety: thread-safety guaranteed by design across the whole pipeline.
+📦 Lightweight: zero-dependency core, ready to run in any Python environment.
+
+### Why not just `ThreadPoolExecutor`?
+
+Fair question — the stdlib solves the *parallelism*, but not the *pipeline*.
+`concurrent.futures` gives you a thread pool; everything around it is on you:
+
+| You need...                                       | With the stdlib             | With DataLoom                       |
+| -------------------------------------------------- | --------------------------- | ----------------------------------- |
+| A continuous producer-consumer flow                 | `Queue` + hand-rolled loops | `Loom` + `Source`                   |
+| Backpressure (producer faster than consumers)       | Manual `Queue(maxsize=...)` | Built-in, configurable              |
+| Clean shutdown (drain queue, close resources)       | Manual sentinels and joins  | `stop()` / `with Loom(...)`         |
+| Workers that survive errors and report them         | try/except in every worker  | Centralized `hooks.on_error`        |
+| Per-item processing metrics                         | Manual instrumentation      | `hooks.on_batch_processed`          |
+| Safe concurrent file writes                         | Manual locking              | Ready-made sinks (JSON, CSV, callback) |
+
+If your use case is "apply a function to a list and collect the results",
+use `ThreadPoolExecutor.map` — it is the right tool. DataLoom is for
+**continuous or long-running flows** where lifecycle, resilience and
+observability matter. And, like every thread-based solution on CPython,
+the parallelism gains apply to **I/O-bound** workloads; for CPU-bound
+work, prefer multiprocessing.
+
+## ✨ Features
+
+- **Cohesive API:** aligned concepts (`Loom`, `Weaver`, `Sink`).
+- **Concurrent:** automatically manages multiple _Weavers_ (threads) in parallel.
+- **Safe:** built-in sinks are thread-safe and exceptions are typed (`LoomError`).
+- **Resilient:** processing errors don't kill the Weavers and are reported through hooks.
+- **Manageable:** `Loom` is a context manager — `with Loom(...) as loom:` guarantees automatic `stop()`.
+- **Backpressure:** the task queue is bounded by default (`queue_maxsize`), preventing unbounded memory growth.
+- **Observable:** lifecycle hooks and per-batch metrics (`on_batch_processed` with result and duration), plus built-in logging.
+
+## 📦 Installation
+
+```bash
+pip install dataloom-engine
+```
+
+The engine core has **zero dependencies**. The demo `RandomNumPySource` and
+`StatisticsProcessor` (used in the quick start below) need NumPy, which
+ships as an optional extra:
+
+```bash
+pip install "dataloom-engine[numpy]"
+```
+
+> ⚠️ **Mind the name:** the package installs the `dataloom_engine` module
+> (`import dataloom_engine`). Don't confuse it with the `dataloom` package
+> on PyPI, which is an ORM by another author and unrelated to this project.
+
+To develop or use the latest version from the repository:
+
+```bash
+git clone https://github.com/dionipadilha/dataloom.git
+cd dataloom
+pip install -e .
+```
+
+## 🚀 Quick Start
+
+Here is a complete example of weaving a data pipeline:
+
+```python
+from pathlib import Path
+import numpy as np
+from dataloom_engine import (
+    Loom,
+    LoomConfig,
+    LoomLogs,
+    Processor,
+    JsonFileSink,
+    ThreadedBufferedSink
+)
+from dataloom_engine.sources import RandomNumPySource
+
+# 1. Define your processing logic (stateless)
+class MyFilterProcessor(Processor):
+    def process(self, batch: np.ndarray) -> dict:
+        # 'batch' is a numpy array sized according to the config
+        avg = float(batch.mean())
+        return {
+            "processed_items": len(batch),
+            "average_value": avg,
+            "status": "high" if avg > 0.5 else "low"
+        }
+
+# 2. Initial setup
+if __name__ == "__main__":
+    # Configure console logging
+    LoomLogs.setup()
+
+    # Engine parameters
+    config = LoomConfig(
+        output_dir=Path("./data_out"),
+        batch_size=100,      # Process 100 items at a time
+        interval_seconds=1   # Produce a new batch every second
+    )
+
+    # 3. Data source
+    # Decoupled from the engine, enabling custom sources (DB, CSV, S3)
+    source = RandomNumPySource(config)
+
+    # 4. Buffered file sink
+    # ThreadedBufferedSink keeps I/O from blocking the processing
+    file_sink = JsonFileSink(config.output_dir)
+    sink = ThreadedBufferedSink(file_sink)
+
+    # 5. Initialize the Loom (the orchestrator)
+    loom = Loom(
+        config=config,
+        processor=MyFilterProcessor(),
+        sink=sink,
+        source=source,
+        num_weavers=4  # 4 threads working in parallel
+    )
+
+    print("🧵 DataLoom started! Press Ctrl+C to stop.")
+    try:
+        # The context manager guarantees stop() and resource cleanup,
+        # even on exceptions or Ctrl+C
+        with loom:
+            loom.start()
+    except KeyboardInterrupt:
+        print("\n🛑 Loom stopped.")
+```
+
+## 🏗️ Architecture
+
+DataLoom is built around a weaving metaphor:
+
+- **Loom:** the main machine. Manages the task queue and the lifecycle.
+- **Weaver:** the worker threads. They take the raw material (a batch), process it and deliver it.
+- **Processor:** the business logic. Turns raw data into information.
+- **Sink:** the final destination. Where the finished product is deposited (e.g. `JsonFileSink`, `CsvFileSink`, or any destination via `CallbackSink`).
+
+## 🛠️ Development and Testing
+
+To contribute to the project or run the test suite:
+
+1.  Install the development dependencies:
+
+    ```bash
+    pip install -e ".[dev]"
+    ```
+
+2.  Run the test suite (via pytest):
+    ```bash
+    pytest
+    ```
+
+3.  Run the linter and type checker:
+    ```bash
+    ruff check .
+    mypy
+    ```
+
+## 📄 License
+
+This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
+
+## 🗒️ Version History
+
+Changes in each release are documented in the [CHANGELOG](CHANGELOG.md).

dataloom_engine-0.4.0/README.md

@@ -0,0 +1,180 @@
+# 🧵 DataLoom
+
+> A lightweight, efficient and safe multi-threaded orchestration engine for Python.
+
+🇧🇷 [Versão em português](README.pt-BR.md)
+
+[![CI](https://github.com/dionipadilha/dataloom/actions/workflows/ci.yml/badge.svg)](https://github.com/dionipadilha/dataloom/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/dataloom-engine)](https://pypi.org/project/dataloom-engine/) [![Python](https://img.shields.io/pypi/pyversions/dataloom-engine)](https://pypi.org/project/dataloom-engine/) ![License](https://img.shields.io/badge/license-MIT-green)
+
+<img width="2752" height="1536" alt="DataLoom: a lightweight, efficient and safe multi-threaded orchestration engine for Python." src="https://github.com/user-attachments/assets/c1f3fc98-5eaf-4add-a38c-2717ae699cd5" />
+
+**DataLoom** is a library designed to process data streams using the **Producer-Consumer** pattern with multiple threads (Weavers). It abstracts away the complexity of queues, locks and lifecycle management, letting you focus purely on your data transformation logic.
+
+## Why DataLoom?
+⚡ Performance: real parallel processing for I/O-bound workloads.
+🧩 Simplicity: an intuitive API inspired by the weaving metaphor.
+🛡️ Safety: thread-safety guaranteed by design across the whole pipeline.
+📦 Lightweight: zero-dependency core, ready to run in any Python environment.
+
+### Why not just `ThreadPoolExecutor`?
+
+Fair question — the stdlib solves the *parallelism*, but not the *pipeline*.
+`concurrent.futures` gives you a thread pool; everything around it is on you:
+
+| You need...                                       | With the stdlib             | With DataLoom                       |
+| -------------------------------------------------- | --------------------------- | ----------------------------------- |
+| A continuous producer-consumer flow                 | `Queue` + hand-rolled loops | `Loom` + `Source`                   |
+| Backpressure (producer faster than consumers)       | Manual `Queue(maxsize=...)` | Built-in, configurable              |
+| Clean shutdown (drain queue, close resources)       | Manual sentinels and joins  | `stop()` / `with Loom(...)`         |
+| Workers that survive errors and report them         | try/except in every worker  | Centralized `hooks.on_error`        |
+| Per-item processing metrics                         | Manual instrumentation      | `hooks.on_batch_processed`          |
+| Safe concurrent file writes                         | Manual locking              | Ready-made sinks (JSON, CSV, callback) |
+
+If your use case is "apply a function to a list and collect the results",
+use `ThreadPoolExecutor.map` — it is the right tool. DataLoom is for
+**continuous or long-running flows** where lifecycle, resilience and
+observability matter. And, like every thread-based solution on CPython,
+the parallelism gains apply to **I/O-bound** workloads; for CPU-bound
+work, prefer multiprocessing.
+
+## ✨ Features
+
+- **Cohesive API:** aligned concepts (`Loom`, `Weaver`, `Sink`).
+- **Concurrent:** automatically manages multiple _Weavers_ (threads) in parallel.
+- **Safe:** built-in sinks are thread-safe and exceptions are typed (`LoomError`).
+- **Resilient:** processing errors don't kill the Weavers and are reported through hooks.
+- **Manageable:** `Loom` is a context manager — `with Loom(...) as loom:` guarantees automatic `stop()`.
+- **Backpressure:** the task queue is bounded by default (`queue_maxsize`), preventing unbounded memory growth.
+- **Observable:** lifecycle hooks and per-batch metrics (`on_batch_processed` with result and duration), plus built-in logging.
+
+## 📦 Installation
+
+```bash
+pip install dataloom-engine
+```
+
+The engine core has **zero dependencies**. The demo `RandomNumPySource` and
+`StatisticsProcessor` (used in the quick start below) need NumPy, which
+ships as an optional extra:
+
+```bash
+pip install "dataloom-engine[numpy]"
+```
+
+> ⚠️ **Mind the name:** the package installs the `dataloom_engine` module
+> (`import dataloom_engine`). Don't confuse it with the `dataloom` package
+> on PyPI, which is an ORM by another author and unrelated to this project.
+
+To develop or use the latest version from the repository:
+
+```bash
+git clone https://github.com/dionipadilha/dataloom.git
+cd dataloom
+pip install -e .
+```
+
+## 🚀 Quick Start
+
+Here is a complete example of weaving a data pipeline:
+
+```python
+from pathlib import Path
+import numpy as np
+from dataloom_engine import (
+    Loom,
+    LoomConfig,
+    LoomLogs,
+    Processor,
+    JsonFileSink,
+    ThreadedBufferedSink
+)
+from dataloom_engine.sources import RandomNumPySource
+
+# 1. Define your processing logic (stateless)
+class MyFilterProcessor(Processor):
+    def process(self, batch: np.ndarray) -> dict:
+        # 'batch' is a numpy array sized according to the config
+        avg = float(batch.mean())
+        return {
+            "processed_items": len(batch),
+            "average_value": avg,
+            "status": "high" if avg > 0.5 else "low"
+        }
+
+# 2. Initial setup
+if __name__ == "__main__":
+    # Configure console logging
+    LoomLogs.setup()
+
+    # Engine parameters
+    config = LoomConfig(
+        output_dir=Path("./data_out"),
+        batch_size=100,      # Process 100 items at a time
+        interval_seconds=1   # Produce a new batch every second
+    )
+
+    # 3. Data source
+    # Decoupled from the engine, enabling custom sources (DB, CSV, S3)
+    source = RandomNumPySource(config)
+
+    # 4. Buffered file sink
+    # ThreadedBufferedSink keeps I/O from blocking the processing
+    file_sink = JsonFileSink(config.output_dir)
+    sink = ThreadedBufferedSink(file_sink)
+
+    # 5. Initialize the Loom (the orchestrator)
+    loom = Loom(
+        config=config,
+        processor=MyFilterProcessor(),
+        sink=sink,
+        source=source,
+        num_weavers=4  # 4 threads working in parallel
+    )
+
+    print("🧵 DataLoom started! Press Ctrl+C to stop.")
+    try:
+        # The context manager guarantees stop() and resource cleanup,
+        # even on exceptions or Ctrl+C
+        with loom:
+            loom.start()
+    except KeyboardInterrupt:
+        print("\n🛑 Loom stopped.")
+```
+
+## 🏗️ Architecture
+
+DataLoom is built around a weaving metaphor:
+
+- **Loom:** the main machine. Manages the task queue and the lifecycle.
+- **Weaver:** the worker threads. They take the raw material (a batch), process it and deliver it.
+- **Processor:** the business logic. Turns raw data into information.
+- **Sink:** the final destination. Where the finished product is deposited (e.g. `JsonFileSink`, `CsvFileSink`, or any destination via `CallbackSink`).
+
+## 🛠️ Development and Testing
+
+To contribute to the project or run the test suite:
+
+1.  Install the development dependencies:
+
+    ```bash
+    pip install -e ".[dev]"
+    ```
+
+2.  Run the test suite (via pytest):
+    ```bash
+    pytest
+    ```
+
+3.  Run the linter and type checker:
+    ```bash
+    ruff check .
+    mypy
+    ```
+
+## 📄 License
+
+This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
+
+## 🗒️ Version History
+
+Changes in each release are documented in the [CHANGELOG](CHANGELOG.md).

{dataloom_engine-0.3.0 → dataloom_engine-0.4.0}/dataloom_engine/__init__.py

@@ -1,11 +1,11 @@
 # dataloom_engine/__init__.py
 
 """
-DataLoom: Um motor de orquestração multi-thread leve e eficiente.
+DataLoom: a lightweight and efficient multi-threaded orchestration engine.
 
-Este módulo expõe a API pública do pacote. Observe que classes internas
-como 'Weaver' não são expostas propositalmente, mantendo a superfície
-de uso limpa e segura para o consumidor.
+This module exposes the package's public API. Note that internal classes
+such as 'Weaver' are deliberately not exposed, keeping the usage surface
+clean and safe for consumers.
 """
 
 from dataloom_engine.config import LoomConfig

dataloom_engine-0.4.0/dataloom_engine/_optional.py

@@ -0,0 +1,25 @@
+# dataloom_engine/_optional.py
+
+"""
+Helpers for optional dependencies.
+The engine core has no third-party requirements; the demo implementations
+that need one import it lazily through these helpers, so that a missing
+dependency fails fast with an actionable message.
+"""
+
+from types import ModuleType
+
+
+def require_numpy(feature: str) -> ModuleType:
+    """
+    Returns the numpy module, or raises an informative ImportError if the
+    optional dependency is not installed.
+    """
+    try:
+        import numpy
+    except ImportError as exc:
+        raise ImportError(
+            f"{feature} requires numpy, which is an optional dependency of "
+            'dataloom-engine. Install it with: pip install "dataloom-engine[numpy]"'
+        ) from exc
+    return numpy

dataloom_engine-0.4.0/dataloom_engine/config.py

@@ -0,0 +1,52 @@
+# dataloom_engine/config.py
+
+"""
+Orchestrator configuration management.
+Centralizes operational parameters such as directories and batch sizes.
+"""
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional, Union
+
+from dataloom_engine.exceptions import ConfigurationError
+
+
+@dataclass
+class LoomConfig:
+    """
+    DataLoom's main configuration object.
+
+    Args:
+        output_dir (Path): Base directory where the built-in Sinks save data.
+            Strings are converted to Path automatically.
+        batch_size (int): Number of items produced per processing cycle.
+        interval_seconds (float): Time interval between task generations.
+        queue_maxsize (Optional[int]): Maximum capacity of the task queue
+            (backpressure). None uses the Loom default (num_weavers * 4);
+            0 means an unbounded queue.
+
+    Raises:
+        ConfigurationError: if any parameter is invalid.
+    """
+
+    output_dir: Union[str, Path]
+    batch_size: int = 10
+    interval_seconds: float = 1.0
+    queue_maxsize: Optional[int] = None
+
+    def __post_init__(self) -> None:
+        self.output_dir = Path(self.output_dir)
+
+        if self.batch_size <= 0:
+            raise ConfigurationError(
+                f"batch_size must be greater than zero (got: {self.batch_size})."
+            )
+        if self.interval_seconds < 0:
+            raise ConfigurationError(
+                f"interval_seconds cannot be negative (got: {self.interval_seconds})."
+            )
+        if self.queue_maxsize is not None and self.queue_maxsize < 0:
+            raise ConfigurationError(
+                f"queue_maxsize cannot be negative (got: {self.queue_maxsize})."
+            )

dataloom_engine-0.4.0/dataloom_engine/exceptions.py

@@ -0,0 +1,25 @@
+# dataloom_engine/exceptions.py
+
+"""
+Custom exceptions for DataLoom.
+Lets consumers catch library-specific errors without relying on
+generic Python exceptions.
+"""
+
+
+class LoomError(Exception):
+    """Base exception for every DataLoom error."""
+
+    pass
+
+
+class ConfigurationError(LoomError):
+    """Raised when LoomConfig validation fails."""
+
+    pass
+
+
+class WeaverError(LoomError):
+    """Raised when a Weaver fails to process a batch."""
+
+    pass

dataloom_engine-0.4.0/dataloom_engine/hooks.py

@@ -0,0 +1,50 @@
+# dataloom_engine/hooks.py
+
+"""
+Observability extension points (Hooks).
+Lets you inject monitoring logic without coupling it to the engine core.
+"""
+
+from typing import Any, Dict
+
+
+class LoomHooks:
+    """
+    Base class for lifecycle hooks.
+    Every method is optional and a no-op by default: override only the
+    points you care about. (Deliberately not an ABC — there is no method
+    you are required to implement.)
+    """
+
+    def on_start(self) -> None:
+        """Called right before the Weavers are started."""
+        pass
+
+    def on_stop(self) -> None:
+        """Called after the orchestrator shuts down gracefully."""
+        pass
+
+    def on_error(self, error: Exception) -> None:
+        """
+        Called when an exception occurs in the Loom's main loop or while
+        processing a batch inside a Weaver (WeaverError).
+
+        Note: it may be invoked from multiple Weaver threads at the same
+        time — implementations must be thread-safe.
+        """
+        pass
+
+    def on_batch_processed(self, result: Dict[str, Any], duration_seconds: float) -> None:
+        """
+        Called after each batch is processed and delivered to the Sink.
+
+        Args:
+            result: The dictionary produced by the Processor for the batch.
+            duration_seconds: Time spent in process() + send(), measured
+                with a monotonic clock.
+
+        Note: like on_error, this is invoked from the Weaver threads —
+        implementations must be thread-safe and fast, since they run on
+        the pipeline's hot path.
+        """
+        pass

{dataloom_engine-0.3.0 → dataloom_engine-0.4.0}/dataloom_engine/logs.py

@@ -1,8 +1,8 @@
 # dataloom_engine/logs.py
 
 """
-Utilitários de configuração de logging para o DataLoom.
-Fornece um namespace estático para configuração rápida.
+Logging configuration utilities for DataLoom.
+Provides a static namespace for quick setup.
 """
 
 import logging
@@ -10,16 +10,16 @@ import logging
 
 class LoomLogs:
     """
-    Namespace utilitário para gerenciamento de logs do DataLoom.
-    Não deve ser instanciado, apenas usado estaticamente.
+    Utility namespace for DataLoom log management.
+    Not meant to be instantiated — use it statically.
     """
 
     @staticmethod
     def setup(level: int = logging.INFO) -> None:
         """
-        Configura o logging básico para stdout.
+        Configures basic logging to stdout.
 
-        Uso:
+        Usage:
             LoomLogs.setup(level=logging.DEBUG)
         """
         logging.basicConfig(