scalable-pypeline 1.2.3__py2.py3-none-any.whl → 2.0.2__py2.py3-none-any.whl

scalable_pypeline-1.2.3.dist-info/METADATA

@@ -1,163 +0,0 @@
-Metadata-Version: 2.1
-Name: scalable-pypeline
-Version: 1.2.3
-Summary: PypeLine - Python pipelines for the Real World
-Home-page: https://gitlab.com/bravos2/pypeline
-Author: Bravos Power Corporation
-License: Apache License 2.0
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: PyYAML (<6,>=5.2)
-Requires-Dist: click (==8.0.4)
-Requires-Dist: marshmallow (<4,>=3.2.1)
-Requires-Dist: requests (>=2.24.0)
-Requires-Dist: redis (<5,>=4.5.4)
-Requires-Dist: rhodb[redis] (<6,>=5.1.1)
-Requires-Dist: attrs (<20,>=19)
-Requires-Dist: boto3 (<2,>=1.11)
-Requires-Dist: croniter (<2,>=1.0.15)
-Requires-Dist: celery-dyrygent (==0.8.0)
-Requires-Dist: itsdangerous (==2.0.1)
-Provides-Extra: build
-Requires-Dist: wheel ; extra == 'build'
-Requires-Dist: twine ; extra == 'build'
-Provides-Extra: dev
-Requires-Dist: honcho (>=1.0.1) ; extra == 'dev'
-Requires-Dist: awscli (>=1.11) ; extra == 'dev'
-Requires-Dist: pylint (>=2.5.3) ; extra == 'dev'
-Requires-Dist: pip-licenses ; extra == 'dev'
-Provides-Extra: flask
-Requires-Dist: Werkzeug (==2.0.3) ; extra == 'flask'
-Requires-Dist: Flask (<2,>=1.1.2) ; extra == 'flask'
-Requires-Dist: flask-smorest (<0.29,>=0.23.0) ; extra == 'flask'
-Requires-Dist: Jinja2 (==3.0.3) ; extra == 'flask'
-Provides-Extra: test
-Requires-Dist: pytest-cov (<3,>=2.6.1) ; extra == 'test'
-Requires-Dist: tox (<4,>=3.14.1) ; extra == 'test'
-Requires-Dist: mock (<2,>=1) ; extra == 'test'
-Requires-Dist: moto (>=1.3.16) ; extra == 'test'
-Requires-Dist: responses (<0.11,>=0.10.16) ; extra == 'test'
-Requires-Dist: fakeredis (<3,>=2.10.3) ; extra == 'test'
-Requires-Dist: importlib-metadata (<5,>=4.12) ; extra == 'test'
-Provides-Extra: web
-Requires-Dist: gunicorn ; extra == 'web'
-Requires-Dist: gevent (<22,>=21.12.0) ; extra == 'web'
-Provides-Extra: workers
-Requires-Dist: celery[redis] (<6,>=5.1.2) ; extra == 'workers'
-Requires-Dist: networkx (>=2.4) ; extra == 'workers'
-
-# PypeLine
-
-## Quickstart
-
-1. Add `pypeline` as a dependency to your Python application
-1. Install extras depending on what you are building:
-
-1. `flask` - Convenient interface for Flask applications
-1. `web` - Some standard web server dependencies we like
-1. `workers` - Installs [Celery](https://docs.celeryproject.org/en/stable/getting-started/introduction.html) and [networkx](https://networkx.org/documentation/stable/index.html), which are required if using pipelines.
-
-## Overview
-
-PypeLines is a fork of [Sermos] (https://gitlab.com/sermos/sermos). PypeLines diverges from Sermos as a SAAS platform and is intented as a suite for job management in conjuction with or indepent from a Flask Web App. Common job management workflow's include running pipelines, scheduled tasks, and other various types of jobs. Pypelines is designed to make these systems faster and more intuitive to create for Python developers.
-
-Under the hood we are simply extending various Celery capabilities like their existing complex workflows and make them suitable for large scale pipelines that can be run in production. To do this PypeLines uses a custom Celery configuration and a library known as [Celery-Dyrygent](https://github.com/ovh/celery-dyrygent) to help orchestrate thousands of tasks at once.
-
-### Pypeline
-
--   Celery Configuration
--   Pipelines
--   CronJobs
--   APIs
--   Utilities
-
-### Your Application
-
-This is where all of your code lives and only has a few _requirements_:
-
-1. It is a base application written in Python.
-1. Scheduled tasks and Pipeline nodes must be Python Methods that accept
-   at least one positional argument: `event`
-1. A `sermos.yaml` file, which is a configuration file for running scheduled tasks and pipelines.
-
-## Celery
-
-Pypelines provides sensical default configurations for the use of
-[Celery](http://www.celeryproject.org/). The default deployment uses RabbitMQ,
-and is recommended. This library can be implemented in any other workflow
-(e.g. Kafka) as desired.
-
-There are two core aspects of Celery that pypeline handles and differ from a
-standard Celery deployment.
-
-### ChainedTask
-
-In `celery.py` when imported it will configure Celery and also run
-`GenerateCeleryTasks().generate()`, which will use the `sermos.yaml` config
-to turn customer methods into decorated Celery tasks.
-
-Part of this process includes adding `ChainedTask` as the _base_ for all of
-these dynamically generated tasks.
-
-`ChainedTask` is a Celery `Task` that injects `tools` and `event` into the
-signature of all dynamically generated tasks.
-
-### SermosScheduler
-
-We allow users to set new scheduled / recurring tasks on-the-fly. Celery's
-default `beat_scheduler` does not support this behavior and would require the
-Beat process be killed/restarted upon every change. Instead, we set our
-custom `sermos.celery_beat:SermosScheduler` as the `beat_scheduler`,
-which takes care of watching the database for new/modified entries and reloads
-dynamically.
-
-## Workers / Tasks / Pipeline Nodes
-
-PypeLine handles decorating the tasks, generating the correct Celery
-chains, etc.
-
-Customer code has one requirement: write a python method that accepts one
-positional argument: `event`
-
-e.g.
-
-    def demo_pipeline_node_a(event):
-        logger.info(f"RUNNING demo_pipeline_node_a: {event}")
-        return
-
-### Generators
-
-_TODO_: This needs to be updated both in code and documentation. Leaving here
-because it's valuable to update in the future.
-
-A common task associated with processing batches of documents is generating
-the list of files to process. `pypeline.generators` contains two helpful
-classes to generate lists of files from S3 and from a local file system.
-
-`S3KeyGenerator` will produce a list of object keys in S3. Example:
-
-    gen = S3KeyGenerator('access-key', 'secret-key')
-    files = gen.list_files(
-        'bucket-name',
-        'folder/path/',
-        offset=0,
-        limit=4,
-        return_full_path=False
-    )
-
-`LocalKeyGenerator` will produce a list of file names on a local file system.
-Example:
-
-    gen = LocalKeyGenerator()
-    files = gen.list_files('/path/to/list/')
-
-## Testing
-
-If you are developing pypeline and want to test this package,
-install the test dependencies:
-
-    $ pip install -e .[test]
-
-Now, run the tests:
-
-    $ tox

scalable_pypeline-1.2.3.dist-info/RECORD

@@ -1,33 +0,0 @@
-pypeline/__init__.py,sha256=NcjoxDLYOmbp4fP2sdOI7frUmFsP_OmIRLppFUOMwJQ,22
-pypeline/celery.py,sha256=rbMCQQqevhdcyFYJSyoQT2b_NAbL3Inc9S789AtzN_w,9038
-pypeline/celery_beat.py,sha256=KNmEpZEFOadVTwtRJtgX0AirSSNq65PFLJer4Hsq-xw,10759
-pypeline/constants.py,sha256=0FLdlV8VZ3FtoTJiriCH8-YQHJuvoSLYYi_4QXHVQYg,4148
-pypeline/extensions.py,sha256=_tv62NtJDrwQUM9mablHUmYyBf8TXlIZmf1LM1rlHtc,599
-pypeline/generators.py,sha256=kRcJCohOMMljar6_nAaICTdNX1cNGyfkSvtl52Pd_hc,6776
-pypeline/logging_config.py,sha256=QbUbSqLtxUOlqLkY0sWxVe9VGFKjghZ-MrlduZmtgLw,5503
-pypeline/pipeline_config_schema.py,sha256=ggVs_cc8KVjKI4jg2TwKqoX82Nt2ET3oUbWk8o1agI4,8900
-pypeline/schedule_config_schema.py,sha256=rGN-tGC9fj-Nc_CVasBUQAOzn2OvYyj0V_S6Az3sq6I,7385
-pypeline/sermos_yaml.py,sha256=A0rFabU0ib9-IY64HZuYvB7sL06sTl84myTYmNSLK3g,17336
-pypeline/flask/__init__.py,sha256=_kh1kQcpuU7yNYH36AtMsaCMYru5l43lYsRBe0lTgKE,523
-pypeline/flask/decorators.py,sha256=zH9OB6DqxLUDSDBMs1Nd3Pt3qYdsAOoD8RL2MliRyRs,2298
-pypeline/flask/flask_sermos.py,sha256=D-mTlJENm8MCYgNmCFwHeZb43XoUBsTO8ER1YqnSL3M,5791
-pypeline/flask/api/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-pypeline/flask/api/pipelines.py,sha256=AT1he7IpykUiDDj7B2tKT0t4wwWQcBZIEeFXWOSHAYU,9026
-pypeline/flask/api/schedules.py,sha256=AHiGYzZL__1sq5KZho75VwPWCn9Pz_ooNM179uKuQ7Q,2314
-pypeline/flask/api/utils.py,sha256=bkUBZIbJHaG9nYahHI2J0NqoqoSlQ6j4aArY9q2ggqE,1302
-pypeline/pipeline/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-pypeline/pipeline/chained_task.py,sha256=G7pakmdUZK29E9X3_I5I2cTxTcEuRlXi_BeDAMJt8Qg,2961
-pypeline/pipeline/generator.py,sha256=Nt65IIs5jc2XCQqWUaOoJIdpaEvQvxJgQSnj9DuYX3s,11509
-pypeline/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-pypeline/utils/config_utils.py,sha256=SwpYy6Xk2apDK2GEb61ctHJh7q8IUo3bHOegE_xsDDk,13029
-pypeline/utils/graph_utils.py,sha256=prs0ATCYCxWGWx8s9_gg3PtwJA1DORIIwUlMfKRjHJM,4642
-pypeline/utils/module_utils.py,sha256=X4O2TdBvvoboK6PxzK18UuKbod9l2BfTIrALk_dI0tM,4166
-pypeline/utils/task_utils.py,sha256=p66d4xcuPuBsqtcnaXgqfNXU3-ZSe4lvN11MkJ5_8XY,22222
-tests/fixtures/__init__.py,sha256=vHbv5BMJXGb6XX764sChg5Ax7fixPuijiYNBuxgVTUQ,41
-tests/fixtures/s3_fixtures.py,sha256=jbsp0WeIibLtjdV1nPSkEuJf1n6e9O7LO-kNFkMqylo,1694
-scalable_pypeline-1.2.3.dist-info/LICENSE,sha256=DVQuDIgE45qn836wDaWnYhSdxoLXgpRRKH4RuTjpRZQ,10174
-scalable_pypeline-1.2.3.dist-info/METADATA,sha256=mt_3JLC-Ux3KNwue-BwQtRIz2ITMz7j2D3x-J999Yj4,6174
-scalable_pypeline-1.2.3.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
-scalable_pypeline-1.2.3.dist-info/entry_points.txt,sha256=ZDh7vdDaHrZD0RwUCiZidXg5-d2fBOYcEo7E6CL4g0U,56
-scalable_pypeline-1.2.3.dist-info/top_level.txt,sha256=C7dpkEOc_-nnsAQb28BfQknjD6XHRyS9ZrvVeoIbV7s,15
-scalable_pypeline-1.2.3.dist-info/RECORD,,

scalable_pypeline-1.2.3.dist-info/entry_points.txt

@@ -1,2 +0,0 @@
-[console_scripts]
-pypeline = pypeline.cli.core:pypeline

tests/fixtures/s3_fixtures.py

@@ -1,52 +0,0 @@
-import os
-import pytest
-import json
-from boto3 import Session
-
-
-@pytest.fixture(scope="session")
-def source_documents(dummy_bucket):
-    session = Session(
-        aws_access_key_id='foo',
-        aws_secret_access_key='bar',
-        region_name='us-east-1'
-    )
-    s3 = session.client('s3')
-    # Add PDFs
-    for f_key in os.listdir('tests/fixtures/pdfs'):
-        if f_key == '.DS_Store':
-            continue
-        with open('tests/fixtures/pdfs/' + f_key, 'rb') as f:
-            object_key = 'pdfs/' + f_key
-            s3.upload_fileobj(f, dummy_bucket.name, object_key)
-
-        # Add 1 hidden file for testing.
-        dstore = 'foo'
-        with open(dstore, 'w') as newfile:
-            newfile.write('bar')
-        s3.upload_file(dstore, dummy_bucket.name, 'pdfs/.DS_Store')
-        os.remove(dstore)
-
-    # Add TIFFs
-    for f_key in os.listdir('tests/fixtures/tiffs'):
-        if f_key == '.DS_Store':
-            continue
-        with open('tests/fixtures/tiffs/' + f_key, 'rb') as f:
-            object_key = 'tiffs/' + f_key
-            s3.upload_fileobj(f, dummy_bucket.name, object_key)
-
-    # Add PNGs
-    for f_key in os.listdir('tests/fixtures/pngs'):
-        if f_key == '.DS_Store':
-            continue
-        with open('tests/fixtures/pngs/' + f_key, 'rb') as f:
-            object_key = 'pngs/' + f_key
-            s3.upload_fileobj(f, dummy_bucket.name, object_key)
-
-    # Add HTML files
-    for f_key in os.listdir('tests/fixtures/html'):
-        if f_key == '.DS_Store':
-            continue
-        with open('tests/fixtures/html/' + f_key, 'rb') as f:
-            object_key = 'html/' + f_key
-            s3.upload_fileobj(f, dummy_bucket.name, object_key)