scalable-pypeline 1.2.3__tar.gz → 2.0.1__tar.gz

scalable-pypeline-2.0.1/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.1
+Name: scalable-pypeline
+Version: 2.0.1
+Summary: PypeLine - Python pipelines for the Real World
+Home-page: https://gitlab.com/bravos2/pypeline
+Author: Bravos Power Corporation
+License: Apache License 2.0
+Description-Content-Type: text/markdown
+Provides-Extra: build
+Provides-Extra: flask
+Provides-Extra: web
+Provides-Extra: workers
+Provides-Extra: dev
+Provides-Extra: test
+License-File: LICENSE
+
+```
+______ __   ________  _____  _     _____  _   _  _____ 
+| ___ \\ \ / /| ___ \|  ___|| |   |_   _|| \ | ||  ___|
+| |_/ / \ V / | |_/ /| |__  | |     | |  |  \| || |__  
+|  __/   \ /  |  __/ |  __| | |     | |  | . ` ||  __| 
+| |      | |  | |    | |___ | |_____| |_ | |\  || |___ 
+\_|      \_/  \_|    \____/ \_____/\___/ \_| \_/\____/                                 
+```
+
+## Overview
+
+PypeLine is a versatile open-source library designed to streamline the management of data workflows and APIs. With PypeLine, you can efficiently schedule cron jobs, execute complex Directed Acyclical Graph (DAG) pipelines, and set up a Flask API complete with OpenAPI documentation.
+
+#### Key Features
+- Cron Job Scheduling: Easily schedule recurring tasks with flexible cron job functionality, ensuring that your processes run reliably at specified intervals.
+- DAG Pipelines: Define and execute DAGs to manage complex data workflows with dependencies. PypeLine handles the execution order and parallelism, ensuring that each task runs in the correct sequence.
+- Flask API with OpenAPI: Quickly configure a RESTful API using Flask, with built-in support for OpenAPI documentation, allowing for clear, standardized documentation of your endpoints.
+
+## Requirements
+
+- RabbitMQ
+- Redis
+- Docker (optional for dev)
+
+## Getting Started
+
+Install PypeLines:
+
+```commandline
+pip install scalable-pypeline[flask,web,workers]>=1.2.3
+```
+
+Configure your Flask project (app.py)
+
+```python
+from flask import Flask
+from pypeline.flask import FlaskPypeline
+from pypeline_demo.api import bp
+from pypeline_demo.config import Config
+from pypeline_demo.extensions import dramatiq
+
+
+
+def create_app():
+    app = Flask(__name__)
+
+    dramatiq.init_app(app)
+
+    # Initialize your app with a configuration
+    app.config.from_object(Config)
+
+    pypeline = FlaskPypeline()
+    pypeline.init_app(app, init_api=True)
+
+    # Register API blueprints you wish 
+    app.extensions["pypeline_core_api"].register_blueprint(bp)
+    # Register application blueprints to application
+    app.register_blueprint(bp)
+
+    return app
+
+
+if __name__ == "__main__":
+    app = create_app()
+    app.run(port=5001)
+```
+
+Configure Dramatiq extension (extensions.py)
+
+```python
+from pypeline.dramatiq import Dramatiq
+
+
+dramatiq = Dramatiq()
+```
+
+Setup your yaml configuration for pypelines (pypeline.yaml)
+
+```yaml
+serviceConfig:
+    - name: pipeline-worker
+      registeredTasks:
+          - handler: pypeline_demo.pipeline.a
+          - handler: pypeline_demo.pipeline.b
+          - handler: pypeline_demo.pipeline.c
+          - handler: pypeline_demo.scheduled_tasks.cron_task
+
+pipelines:
+    demo_pipeline:
+        name: Demo Pipeline
+        description: Pipeline to show examples of DAG Adjacency
+        schemaVersion: 1
+        config:
+            dagAdjacency:
+                a:
+                    - b
+                    - c
+            metadata:
+                maxRetry: 1
+                retryBackoff: 180
+                retryBackoffMax: 300
+                retryJitter: true
+                maxTtl: 10800
+                queue: new-queue
+            taskDefinitions:
+                a:
+                    handler: pypeline_demo.pipeline.a
+                b:
+                    handler:  pypeline_demo.pipeline.b
+                c:
+                    handler:  pypeline_demo.pipeline.c
+scheduledTasks:
+    cron-task:
+        name: Example cron task
+        enabled: true
+        config:
+            task: pypeline_demo.scheduled_tasks.cron_task
+            queue: new-queue
+            schedule:
+                minute: '*'
+                hour: '*'
+                dayOfWeek: '*'
+                dayOfMonth: '*'
+                monthOfYear: '*'
+        schemaVersion: 1
+```
+
+Setup your modules to be executed by yaml (pipeline.py && scheduled_tasks.py)
+
+```python
+import time
+
+
+def a(event):
+    print("A")
+
+
+def b(event):
+    print("B")
+    time.sleep(10)
+
+
+def c(event):
+    print("C")
+```
+
+```python
+def cron_task():
+    print("HI")
+```
+
+Configure your environment variables (demo.env)
+
+```env
+SERMOS_BASE_URL=local
+PYPELINE_CLIENT_PKG_NAME=pypeline_demo
+REDIS_URL=redis://:password@localhost:6379/0
+RABBITMQ_URL=amqp://admin:password@localhost:5672
+```
+
+Start Rabbit & Redis as your message broker and backend results storage.  We use `docker compose` for this.
+
+## DEMO PROJECT COMING SOON!
+
+
+## Testing
+
+If you are developing pypeline and want to test this package,
+install the test dependencies:
+
+    $ pip install -e .[test]
+
+Now, run the tests:
+
+    $ tox

scalable-pypeline-2.0.1/README.md

@@ -0,0 +1,175 @@
+```
+______ __   ________  _____  _     _____  _   _  _____ 
+| ___ \\ \ / /| ___ \|  ___|| |   |_   _|| \ | ||  ___|
+| |_/ / \ V / | |_/ /| |__  | |     | |  |  \| || |__  
+|  __/   \ /  |  __/ |  __| | |     | |  | . ` ||  __| 
+| |      | |  | |    | |___ | |_____| |_ | |\  || |___ 
+\_|      \_/  \_|    \____/ \_____/\___/ \_| \_/\____/                                 
+```
+
+## Overview
+
+PypeLine is a versatile open-source library designed to streamline the management of data workflows and APIs. With PypeLine, you can efficiently schedule cron jobs, execute complex Directed Acyclical Graph (DAG) pipelines, and set up a Flask API complete with OpenAPI documentation.
+
+#### Key Features
+- Cron Job Scheduling: Easily schedule recurring tasks with flexible cron job functionality, ensuring that your processes run reliably at specified intervals.
+- DAG Pipelines: Define and execute DAGs to manage complex data workflows with dependencies. PypeLine handles the execution order and parallelism, ensuring that each task runs in the correct sequence.
+- Flask API with OpenAPI: Quickly configure a RESTful API using Flask, with built-in support for OpenAPI documentation, allowing for clear, standardized documentation of your endpoints.
+
+## Requirements
+
+- RabbitMQ
+- Redis
+- Docker (optional for dev)
+
+## Getting Started
+
+Install PypeLines:
+
+```commandline
+pip install scalable-pypeline[flask,web,workers]>=1.2.3
+```
+
+Configure your Flask project (app.py)
+
+```python
+from flask import Flask
+from pypeline.flask import FlaskPypeline
+from pypeline_demo.api import bp
+from pypeline_demo.config import Config
+from pypeline_demo.extensions import dramatiq
+
+
+
+def create_app():
+    app = Flask(__name__)
+
+    dramatiq.init_app(app)
+
+    # Initialize your app with a configuration
+    app.config.from_object(Config)
+
+    pypeline = FlaskPypeline()
+    pypeline.init_app(app, init_api=True)
+
+    # Register API blueprints you wish 
+    app.extensions["pypeline_core_api"].register_blueprint(bp)
+    # Register application blueprints to application
+    app.register_blueprint(bp)
+
+    return app
+
+
+if __name__ == "__main__":
+    app = create_app()
+    app.run(port=5001)
+```
+
+Configure Dramatiq extension (extensions.py)
+
+```python
+from pypeline.dramatiq import Dramatiq
+
+
+dramatiq = Dramatiq()
+```
+
+Setup your yaml configuration for pypelines (pypeline.yaml)
+
+```yaml
+serviceConfig:
+    - name: pipeline-worker
+      registeredTasks:
+          - handler: pypeline_demo.pipeline.a
+          - handler: pypeline_demo.pipeline.b
+          - handler: pypeline_demo.pipeline.c
+          - handler: pypeline_demo.scheduled_tasks.cron_task
+
+pipelines:
+    demo_pipeline:
+        name: Demo Pipeline
+        description: Pipeline to show examples of DAG Adjacency
+        schemaVersion: 1
+        config:
+            dagAdjacency:
+                a:
+                    - b
+                    - c
+            metadata:
+                maxRetry: 1
+                retryBackoff: 180
+                retryBackoffMax: 300
+                retryJitter: true
+                maxTtl: 10800
+                queue: new-queue
+            taskDefinitions:
+                a:
+                    handler: pypeline_demo.pipeline.a
+                b:
+                    handler:  pypeline_demo.pipeline.b
+                c:
+                    handler:  pypeline_demo.pipeline.c
+scheduledTasks:
+    cron-task:
+        name: Example cron task
+        enabled: true
+        config:
+            task: pypeline_demo.scheduled_tasks.cron_task
+            queue: new-queue
+            schedule:
+                minute: '*'
+                hour: '*'
+                dayOfWeek: '*'
+                dayOfMonth: '*'
+                monthOfYear: '*'
+        schemaVersion: 1
+```
+
+Setup your modules to be executed by yaml (pipeline.py && scheduled_tasks.py)
+
+```python
+import time
+
+
+def a(event):
+    print("A")
+
+
+def b(event):
+    print("B")
+    time.sleep(10)
+
+
+def c(event):
+    print("C")
+```
+
+```python
+def cron_task():
+    print("HI")
+```
+
+Configure your environment variables (demo.env)
+
+```env
+SERMOS_BASE_URL=local
+PYPELINE_CLIENT_PKG_NAME=pypeline_demo
+REDIS_URL=redis://:password@localhost:6379/0
+RABBITMQ_URL=amqp://admin:password@localhost:5672
+```
+
+Start Rabbit & Redis as your message broker and backend results storage.  We use `docker compose` for this.
+
+## DEMO PROJECT COMING SOON!
+
+
+## Testing
+
+If you are developing pypeline and want to test this package,
+install the test dependencies:
+
+    $ pip install -e .[test]
+
+Now, run the tests:
+
+    $ tox

scalable-pypeline-2.0.1/pypeline/__init__.py

@@ -0,0 +1 @@
+__version__ = "2.0.1"

scalable-pypeline-2.0.1/pypeline/barrier.py

@@ -0,0 +1,34 @@
+import time
+
+import redis
+
+
+class LockingParallelBarrier:
+    def __init__(self, redis_url, task_key="task_counter", lock_key="task_lock"):
+        # Connect to Redis using the provided URL
+        self.redis = redis.StrictRedis.from_url(redis_url, decode_responses=True)
+        self.task_key = task_key
+        self.lock_key = lock_key
+
+    def acquire_lock(self, timeout=5):
+        """Acquire a lock using Redis."""
+        while True:
+            if self.redis.set(self.lock_key, "locked", nx=True, ex=timeout):
+                return True
+            time.sleep(0.1)
+
+    def release_lock(self):
+        """Release the lock in Redis."""
+        self.redis.delete(self.lock_key)
+
+    def set_task_count(self, count):
+        """Initialize the task counter in Redis."""
+        self.redis.set(self.task_key, count)
+
+    def decrement_task_count(self):
+        """Decrement the task counter in Redis."""
+        return self.redis.decr(self.task_key)
+
+    def get_task_count(self):
+        """Get the current value of the task counter."""
+        return int(self.redis.get(self.task_key) or 0)