pipefy 0.0.1__tar.gz

pipefy-0.0.1/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2024 [Daniel Naumowich]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

pipefy-0.0.1/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.1
+Name: pipefy
+Version: 0.0.1
+Summary: 
+Author: Daniel Naumowich
+Author-email: cskteam4@gmail.com
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: chardet (>=5.2.0,<6.0.0)
+Requires-Dist: httpx (>=0.27.0,<0.28.0)
+Requires-Dist: loguru (>=0.7.2,<0.8.0)
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Requires-Dist: pytest (>=8.2.2,<9.0.0)
+Requires-Dist: setuptools (>=75.6.0,<76.0.0)
+Description-Content-Type: text/markdown
+
+# ETL Pipeline Library
+
+## Description
+
+This library is designed for building **ETL pipelines** (Extract, Transform, Load). It provides a set of processors to perform various data operations, such as extracting, transforming, and loading data into the desired format. Each processor in the library is a standalone unit that can be linked together in a chain to perform complex operations.
+
+Processors support working with various data formats (CSV, JSON, ZIP, and others), allow flexible data processing configurations, handle exceptions, and interact with file systems to read and write data.
+
+## Key Features
+
+- **Clean architecture with processor chains**: Each processor performs a specific task and can pass data to the next processor in the chain.
+- **Support for various data formats**: The library includes processors for working with CSV, JSON, ZIP, and other formats.
+- **Flexibility and configuration**: Support for various configuration options, such as custom converters for numbers, error handling methods, and working with large files.
+- **Automation**: Option to automatically delete the source files after processing.
+- **Error handling mechanisms**: The library provides support for ignoring or handling different exceptions at various stages of data processing.
+
+## Main Components
+
+### 1. **BaseProcessor**
+The base class for all processors. It supports the chain of responsibility, allows configuring subprocessors for parallel or sequential processing, and manages exceptions.
+
+### 2. **ChainAnyProcessor**
+A processor that attempts to process data through a series of sub-processors in sequence. 
+
+- **Behavior**:
+  - Each sub-processor tries to handle the data.
+  - The first successful sub-processor is moved to the front of the chain for prioritization in future attempts.
+  - If all sub-processors fail, an error is logged.
+- **Use Case**: Useful when there are multiple processors capable of handling the same type of data, but with varying likelihoods of success.
+
+### 3. **ChainAllProcessor**
+A processor that runs multiple sub-processors in parallel, either using threads or processes.
+
+- **Behavior**:
+  - Executes all sub-processors simultaneously.
+  - Yields results from sub-processors as soon as they become available.
+  - Logs any exceptions that occur during processing.
+- **Use Case**: Ideal for parallelizing independent operations, such as processing different parts of a dataset.
+
+### 4. **HttpDataExtractProcessor**
+A processor for extracting data via HTTP GET requests, with the option to save the retrieved data to a file.
+
+### 5. **HttpxStreamDownloadProcessor**
+A processor for streaming large file downloads via HTTP, with the ability to process data in parallel.
+
+### 6. **CsvParser**
+A processor for parsing CSV files using the `pandas` library. It supports numerous options, including handling delimiters, skipping empty lines, and processing data in chunks.
+
+### 7. **JsonParser**
+A processor for parsing JSON data from strings or files into Python dictionaries. It supports configuring how numbers, constants (e.g., NaN), and JSON objects are handled with custom functions.
+
+### 8. **BaseUnzipProcessor**
+A processor for extracting files from `.zip` archives. It supports extracting data into files, managing chunk sizes, and deleting the original archive after processing.
+
+### 9. **RetryProcessor**
+
+A processor designed to handle transient errors by retrying operations based on a customizable retry policy.
+
+- **Key Features**:
+  - **Configurable Retry Logic**: Supports setting a maximum number of retries and delays between attempts (fixed or exponential backoff).
+  - **White-listed Exceptions**: Retries only specific exceptions defined by the user.
+  - **Logging and Metrics**: Logs every retry attempt and provides details about failures.
+  - **Fallback Handling**: Allows defining a fallback action if all retry attempts fail.
+- **Use Case**: Ideal for handling temporary issues like network timeouts, database connection errors, or transient API failures.
+
+## Example Usage
+
+### Example ETL Pipeline(Flow 1)
+
+```python
+from processors.http import HttpDataExtractProcessor
+from processors.csv_parser import CsvParser
+from processors.json_parser import JsonParser
+from processors.unzip import BaseUnzipProcessor
+from processors.file_system import AbstractFileSystemManager
+
+# Create file system manager
+file_system_manager = AbstractFileSystemManager()
+
+# Create processors
+http_processor = HttpDataExtractProcessor(file_system_manager)
+csv_parser = CsvParser(file_system_manager)
+json_parser = JsonParser(file_system_manager)
+unzip_processor = BaseUnzipProcessor(file_system_manager)
+
+# Create processor chain
+http_processor.set_next(csv_parser).set_next(json_parser).set_next(unzip_processor)
+
+# Run the pipeline
+input_data = "https://example.com/data.zip"
+for output in http_processor.process(input_data):
+    print(output)
+```
+
+### Example ETL Pipeline(Flow 2: Using Operator Overloads)
+
+```python
+from processors.http import HttpDataExtractProcessor
+from processors.csv_parser import CsvParser
+from processors.json_parser import JsonParser
+from processors.unzip import BaseUnzipProcessor
+from processors.file_system import AbstractFileSystemManager
+
+# Create file system manager
+file_system_manager = AbstractFileSystemManager()
+
+# Create processors
+http_processor = HttpDataExtractProcessor(file_system_manager)
+csv_parser = CsvParser(file_system_manager)
+json_parser = JsonParser(file_system_manager)
+unzip_processor = BaseUnzipProcessor(file_system_manager)
+
+# Create processor chain using `>>`
+pipeline = http_processor >> csv_parser >> json_parser >> unzip_processor
+
+# Run the pipeline
+input_data = "https://example.com/data.zip"
+for output in pipeline.process(input_data):
+    print(output)
+```
+
+### Example ETL Pipeline(Flow 3: Combining Parallel and Sequential Processing)
+
+```python
+from operations import Op, OpAll, OpAny, PipelineConverter
+from processors.file_system import FileModeEnum
+
+def main():
+    # Define the pipeline with parallel processing for parsing and unzipping
+    pipeline = [
+        Op(
+            BaseOpEnum.downloading, 
+            kwargs={
+                "file_system_manager": {...},
+                "retry_args": {
+                    "max_retries": 3,
+                    "delay": 1,
+                    "delay_step": 1,
+                    "retry_with_white_exc": True,
+                },
+            }
+        ),
+        Op(
+            BaseOpEnum.upzip,
+            kwargs={
+                "file_system_manager": {...}
+            }
+        ),
+        OpAny(
+            operations = [
+                Op(
+                    BaseOpEnum.json_parsing, 
+                    kwargs={
+                        "file_system_manager": {...}
+                    }
+                ),
+                Op(
+                    BaseOpEnum.csv_parsing, 
+                    kwargs={
+                        "file_system_manager": {...},
+                        "white_exceptions": [
+                            TypeError,
+                            BaseExceptionsEnum.csv_parsing_error,
+                            BaseExceptionsEnum.file_system_exception,
+                        ],
+                    }
+                ),
+            ],
+            kwargs = {...}
+        ),
+        Op(
+            "app.processor.CustomProcessor",
+            kwargs = {...}
+        ),
+    ]
+
+    # Convert the high-level pipeline definition into a processor chain
+    processor = PipelineConverter(pipeline=pipeline).convert()
+
+    # Execute the pipeline
+    input_data = "https://example.com/data.zip"
+    for output in processor.process(input_data):
+        print(output)
+
+if __name__ == "__main__":
+    main()
+```
+
+## Extensibility
+The library is developer-friendly, making it easy to extend and integrate custom processors. Adding a new processor involves subclassing BaseProcessor and implementing its process method. This design ensures seamless integration with existing pipelines.
+
+## Example Usage
+
+```python
+from processors.base import BaseProcessor
+
+class CustomProcessor(BaseProcessor):
+    def process(self, input_data):
+        # Custom data processing logic
+        yield transformed_data
+```
+
+This library is designed to handle a wide range of ETL requirements, making it a reliable choice for both simple and complex workflows. Whether you’re dealing with small-scale data extraction or managing high-throughput pipelines, the library’s flexibility, extensibility, and robust error handling provide a solid foundation for your ETL needs.

pipefy-0.0.1/README.md

@@ -0,0 +1,203 @@
+# ETL Pipeline Library
+
+## Description
+
+This library is designed for building **ETL pipelines** (Extract, Transform, Load). It provides a set of processors to perform various data operations, such as extracting, transforming, and loading data into the desired format. Each processor in the library is a standalone unit that can be linked together in a chain to perform complex operations.
+
+Processors support working with various data formats (CSV, JSON, ZIP, and others), allow flexible data processing configurations, handle exceptions, and interact with file systems to read and write data.
+
+## Key Features
+
+- **Clean architecture with processor chains**: Each processor performs a specific task and can pass data to the next processor in the chain.
+- **Support for various data formats**: The library includes processors for working with CSV, JSON, ZIP, and other formats.
+- **Flexibility and configuration**: Support for various configuration options, such as custom converters for numbers, error handling methods, and working with large files.
+- **Automation**: Option to automatically delete the source files after processing.
+- **Error handling mechanisms**: The library provides support for ignoring or handling different exceptions at various stages of data processing.
+
+## Main Components
+
+### 1. **BaseProcessor**
+The base class for all processors. It supports the chain of responsibility, allows configuring subprocessors for parallel or sequential processing, and manages exceptions.
+
+### 2. **ChainAnyProcessor**
+A processor that attempts to process data through a series of sub-processors in sequence. 
+
+- **Behavior**:
+  - Each sub-processor tries to handle the data.
+  - The first successful sub-processor is moved to the front of the chain for prioritization in future attempts.
+  - If all sub-processors fail, an error is logged.
+- **Use Case**: Useful when there are multiple processors capable of handling the same type of data, but with varying likelihoods of success.
+
+### 3. **ChainAllProcessor**
+A processor that runs multiple sub-processors in parallel, either using threads or processes.
+
+- **Behavior**:
+  - Executes all sub-processors simultaneously.
+  - Yields results from sub-processors as soon as they become available.
+  - Logs any exceptions that occur during processing.
+- **Use Case**: Ideal for parallelizing independent operations, such as processing different parts of a dataset.
+
+### 4. **HttpDataExtractProcessor**
+A processor for extracting data via HTTP GET requests, with the option to save the retrieved data to a file.
+
+### 5. **HttpxStreamDownloadProcessor**
+A processor for streaming large file downloads via HTTP, with the ability to process data in parallel.
+
+### 6. **CsvParser**
+A processor for parsing CSV files using the `pandas` library. It supports numerous options, including handling delimiters, skipping empty lines, and processing data in chunks.
+
+### 7. **JsonParser**
+A processor for parsing JSON data from strings or files into Python dictionaries. It supports configuring how numbers, constants (e.g., NaN), and JSON objects are handled with custom functions.
+
+### 8. **BaseUnzipProcessor**
+A processor for extracting files from `.zip` archives. It supports extracting data into files, managing chunk sizes, and deleting the original archive after processing.
+
+### 9. **RetryProcessor**
+
+A processor designed to handle transient errors by retrying operations based on a customizable retry policy.
+
+- **Key Features**:
+  - **Configurable Retry Logic**: Supports setting a maximum number of retries and delays between attempts (fixed or exponential backoff).
+  - **White-listed Exceptions**: Retries only specific exceptions defined by the user.
+  - **Logging and Metrics**: Logs every retry attempt and provides details about failures.
+  - **Fallback Handling**: Allows defining a fallback action if all retry attempts fail.
+- **Use Case**: Ideal for handling temporary issues like network timeouts, database connection errors, or transient API failures.
+
+## Example Usage
+
+### Example ETL Pipeline(Flow 1)
+
+```python
+from processors.http import HttpDataExtractProcessor
+from processors.csv_parser import CsvParser
+from processors.json_parser import JsonParser
+from processors.unzip import BaseUnzipProcessor
+from processors.file_system import AbstractFileSystemManager
+
+# Create file system manager
+file_system_manager = AbstractFileSystemManager()
+
+# Create processors
+http_processor = HttpDataExtractProcessor(file_system_manager)
+csv_parser = CsvParser(file_system_manager)
+json_parser = JsonParser(file_system_manager)
+unzip_processor = BaseUnzipProcessor(file_system_manager)
+
+# Create processor chain
+http_processor.set_next(csv_parser).set_next(json_parser).set_next(unzip_processor)
+
+# Run the pipeline
+input_data = "https://example.com/data.zip"
+for output in http_processor.process(input_data):
+    print(output)
+```
+
+### Example ETL Pipeline(Flow 2: Using Operator Overloads)
+
+```python
+from processors.http import HttpDataExtractProcessor
+from processors.csv_parser import CsvParser
+from processors.json_parser import JsonParser
+from processors.unzip import BaseUnzipProcessor
+from processors.file_system import AbstractFileSystemManager
+
+# Create file system manager
+file_system_manager = AbstractFileSystemManager()
+
+# Create processors
+http_processor = HttpDataExtractProcessor(file_system_manager)
+csv_parser = CsvParser(file_system_manager)
+json_parser = JsonParser(file_system_manager)
+unzip_processor = BaseUnzipProcessor(file_system_manager)
+
+# Create processor chain using `>>`
+pipeline = http_processor >> csv_parser >> json_parser >> unzip_processor
+
+# Run the pipeline
+input_data = "https://example.com/data.zip"
+for output in pipeline.process(input_data):
+    print(output)
+```
+
+### Example ETL Pipeline(Flow 3: Combining Parallel and Sequential Processing)
+
+```python
+from operations import Op, OpAll, OpAny, PipelineConverter
+from processors.file_system import FileModeEnum
+
+def main():
+    # Define the pipeline with parallel processing for parsing and unzipping
+    pipeline = [
+        Op(
+            BaseOpEnum.downloading, 
+            kwargs={
+                "file_system_manager": {...},
+                "retry_args": {
+                    "max_retries": 3,
+                    "delay": 1,
+                    "delay_step": 1,
+                    "retry_with_white_exc": True,
+                },
+            }
+        ),
+        Op(
+            BaseOpEnum.upzip,
+            kwargs={
+                "file_system_manager": {...}
+            }
+        ),
+        OpAny(
+            operations = [
+                Op(
+                    BaseOpEnum.json_parsing, 
+                    kwargs={
+                        "file_system_manager": {...}
+                    }
+                ),
+                Op(
+                    BaseOpEnum.csv_parsing, 
+                    kwargs={
+                        "file_system_manager": {...},
+                        "white_exceptions": [
+                            TypeError,
+                            BaseExceptionsEnum.csv_parsing_error,
+                            BaseExceptionsEnum.file_system_exception,
+                        ],
+                    }
+                ),
+            ],
+            kwargs = {...}
+        ),
+        Op(
+            "app.processor.CustomProcessor",
+            kwargs = {...}
+        ),
+    ]
+
+    # Convert the high-level pipeline definition into a processor chain
+    processor = PipelineConverter(pipeline=pipeline).convert()
+
+    # Execute the pipeline
+    input_data = "https://example.com/data.zip"
+    for output in processor.process(input_data):
+        print(output)
+
+if __name__ == "__main__":
+    main()
+```
+
+## Extensibility
+The library is developer-friendly, making it easy to extend and integrate custom processors. Adding a new processor involves subclassing BaseProcessor and implementing its process method. This design ensures seamless integration with existing pipelines.
+
+## Example Usage
+
+```python
+from processors.base import BaseProcessor
+
+class CustomProcessor(BaseProcessor):
+    def process(self, input_data):
+        # Custom data processing logic
+        yield transformed_data
+```
+
+This library is designed to handle a wide range of ETL requirements, making it a reliable choice for both simple and complex workflows. Whether you’re dealing with small-scale data extraction or managing high-throughput pipelines, the library’s flexibility, extensibility, and robust error handling provide a solid foundation for your ETL needs.

pipefy-0.0.1/pipefy/etl/extract/http/__init__.py

@@ -0,0 +1,9 @@
+from pipefy.etl.extract.http.http_extractor import (
+    HttpDataExtractProcessor,
+    HttpxStreamDownloadProcessor,
+)
+
+__all__ = (
+    "HttpDataExtractProcessor",
+    "HttpxStreamDownloadProcessor",
+)