morp 5.2.1__tar.gz → 7.0.0__tar.gz

morp-7.0.0/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: morp
+Version: 7.0.0
+Summary: Send and receive messages without thinking about it
+Author-email: Jay Marcyes <jay@marcyes.com>
+License-Expression: MIT
+Project-URL: Homepage, http://github.com/Jaymon/morp
+Project-URL: Repository, https://github.com/Jaymon/morp
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Utilities
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: dsnparse
+Requires-Dist: datatypes
+Provides-Extra: tests
+Requires-Dist: testdata; extra == "tests"
+Provides-Extra: postgres
+Requires-Dist: psycopg; extra == "postgres"
+Provides-Extra: sqs
+Requires-Dist: boto3; extra == "sqs"
+Provides-Extra: encryption
+Requires-Dist: cryptography; extra == "encryption"
+Dynamic: license-file
+
+# Morp
+
+Simple message processing without really thinking about it. Morp can use dropfiles (simple text files), Postgres, and [Amazon SQS](http://aws.amazon.com/sqs/).
+
+
+## Installation
+
+Use pip to install the latest stable version:
+
+    pip install morp
+    
+Morp only supports the dropfiles interface out of the box, you'll need to install certain dependencies depending on what interface you want to use:
+
+    pip install morp[sqs]
+    pip install morp[postgres]
+    
+To install the development version:
+
+    pip install -U "git+https://github.com/Jaymon/morp#egg=morp"
+
+
+## 1 Minute Getting Started
+
+Send and receive a `Foo` message.
+
+First, let's set our environment variable to use dropfiles (local files suitable for development and prototyping) interface:
+
+    export MORP_DSN=dropfile:${TMPDIR}
+
+Then, let's create three files in our working directory:
+
+* `tasks.py` - We'll define our `Message` classes here.
+* `send.py` - We'll send messages from this script.
+* `recv.py` - We'll receive messages from this script.
+
+
+Let's create our `Message` class in `tasks.py`:
+
+```python
+# tasks.py
+from morp import Message
+
+class Foo(Message):
+    some_field: int
+    some_other_field: str
+    
+    def handle(self):
+        # this will be run when a Foo message is consumed
+        print(self.fields)
+```
+
+Now, let's flesh out our `recv.py` file:
+
+```python
+# recv.py
+
+import asyncio
+
+# import our Foo message class from our tasks.py file
+from tasks import Foo
+
+# Foo's `process` method will call `Foo.handle` for each Foo instance received
+asyncio.run(Foo.process())
+```
+
+And start it up:
+
+```
+$ python recv.py
+```
+
+
+Finally, let's send some messages by fleshing out `send.py`:
+
+```python
+# send.py
+
+import asyncio
+
+from tasks import Foo
+
+async def send_messages():
+    # create a message and send it manually
+    f = Foo()
+    f.some_field = 1
+    f.some_other_field = "one"
+    f.ignored_field = True
+    await f.send()
+
+    # quickly send a message
+    await Foo.create(
+        some_field=2,
+        some_other_field="two",
+    )
+    
+asyncio.run(send_messages())
+```
+
+And running it in a separate shell from the shell running our `recv.py` script (it should send two messages):
+
+```
+$ python send.py
+```
+
+That's it! Our running `recv.py` script should've received the messages we sent when we ran our `send.py` script.
+
+
+## DSN
+
+You configure your connection using a dsn in the form:
+
+    InterfaceName://username:password@host:port/path?param1=value1&param2=value2
+
+So, to connect to [Amazon SQS](http://aws.amazon.com/sqs/), you would do:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@
+
+You can also override some default values like `region` and `read_lock`:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@?region=${AWS_DEFAULT_REGION}&read_lock=120
+
+
+### Serializers
+
+* `pickle` (default)
+* `json`
+
+```
+MORP_DSN="sqs://x:x@?serializer=json"
+```
+
+
+## Encryption
+
+You might need to install some dependencies:
+
+```
+pip install morp[encryption]
+```
+
+If you would like to encrypt all your messages, you can pass in a `key` argument to your DSN and Morp will take care of encrypting and decrypting the messages for you transparently.
+
+Let's just modify our DSN to pass in our key:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@?key=jy4XWRuEsrH98RD2VeLG62uVLCPWpdUh
+
+That's it, every message will now be encrypted on send and decrypted on receive. If you're using SQS you can also use [Amazon's key management service](https://github.com/Jaymon/morp/blob/master/docs/KMS.md) to handle the encryption for you.
+
+
+## Environment configuration
+
+### MORP_DISABLED
+
+By default every message will be sent, if you just want to test functionality without actually sending the message you can set this environment variable to turn off all the queues.
+
+    MORP_DISABLED = 1 # queue is off
+    MORP_DISABLED = 0 # queue is on
+
+
+### MORP_PREFIX
+
+If you would like to have your queue names prefixed with something (eg, `prod` or `dev`) then you can set this environment variable and it will be prefixed to the queue name.
+
+
+### MORP_DSN
+
+Set this environment variable with your connection DSN so Morp can automatically configure itself when the interface is first requested.
+
+
+## FAQ
+
+### I would like to have multiple queues
+
+By default, Morp will send any message from any `morp.Message` derived class to `Message.get_name()`, you can override this behavior by giving your child class a `._name` property:
+
+```python
+from morp import Message
+
+class childMsg(Message):
+    _name = "custom-queue-name"
+```

morp-7.0.0/README.md

@@ -0,0 +1,181 @@
+# Morp
+
+Simple message processing without really thinking about it. Morp can use dropfiles (simple text files), Postgres, and [Amazon SQS](http://aws.amazon.com/sqs/).
+
+
+## Installation
+
+Use pip to install the latest stable version:
+
+    pip install morp
+    
+Morp only supports the dropfiles interface out of the box, you'll need to install certain dependencies depending on what interface you want to use:
+
+    pip install morp[sqs]
+    pip install morp[postgres]
+    
+To install the development version:
+
+    pip install -U "git+https://github.com/Jaymon/morp#egg=morp"
+
+
+## 1 Minute Getting Started
+
+Send and receive a `Foo` message.
+
+First, let's set our environment variable to use dropfiles (local files suitable for development and prototyping) interface:
+
+    export MORP_DSN=dropfile:${TMPDIR}
+
+Then, let's create three files in our working directory:
+
+* `tasks.py` - We'll define our `Message` classes here.
+* `send.py` - We'll send messages from this script.
+* `recv.py` - We'll receive messages from this script.
+
+
+Let's create our `Message` class in `tasks.py`:
+
+```python
+# tasks.py
+from morp import Message
+
+class Foo(Message):
+    some_field: int
+    some_other_field: str
+    
+    def handle(self):
+        # this will be run when a Foo message is consumed
+        print(self.fields)
+```
+
+Now, let's flesh out our `recv.py` file:
+
+```python
+# recv.py
+
+import asyncio
+
+# import our Foo message class from our tasks.py file
+from tasks import Foo
+
+# Foo's `process` method will call `Foo.handle` for each Foo instance received
+asyncio.run(Foo.process())
+```
+
+And start it up:
+
+```
+$ python recv.py
+```
+
+
+Finally, let's send some messages by fleshing out `send.py`:
+
+```python
+# send.py
+
+import asyncio
+
+from tasks import Foo
+
+async def send_messages():
+    # create a message and send it manually
+    f = Foo()
+    f.some_field = 1
+    f.some_other_field = "one"
+    f.ignored_field = True
+    await f.send()
+
+    # quickly send a message
+    await Foo.create(
+        some_field=2,
+        some_other_field="two",
+    )
+    
+asyncio.run(send_messages())
+```
+
+And running it in a separate shell from the shell running our `recv.py` script (it should send two messages):
+
+```
+$ python send.py
+```
+
+That's it! Our running `recv.py` script should've received the messages we sent when we ran our `send.py` script.
+
+
+## DSN
+
+You configure your connection using a dsn in the form:
+
+    InterfaceName://username:password@host:port/path?param1=value1&param2=value2
+
+So, to connect to [Amazon SQS](http://aws.amazon.com/sqs/), you would do:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@
+
+You can also override some default values like `region` and `read_lock`:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@?region=${AWS_DEFAULT_REGION}&read_lock=120
+
+
+### Serializers
+
+* `pickle` (default)
+* `json`
+
+```
+MORP_DSN="sqs://x:x@?serializer=json"
+```
+
+
+## Encryption
+
+You might need to install some dependencies:
+
+```
+pip install morp[encryption]
+```
+
+If you would like to encrypt all your messages, you can pass in a `key` argument to your DSN and Morp will take care of encrypting and decrypting the messages for you transparently.
+
+Let's just modify our DSN to pass in our key:
+
+    sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@?key=jy4XWRuEsrH98RD2VeLG62uVLCPWpdUh
+
+That's it, every message will now be encrypted on send and decrypted on receive. If you're using SQS you can also use [Amazon's key management service](https://github.com/Jaymon/morp/blob/master/docs/KMS.md) to handle the encryption for you.
+
+
+## Environment configuration
+
+### MORP_DISABLED
+
+By default every message will be sent, if you just want to test functionality without actually sending the message you can set this environment variable to turn off all the queues.
+
+    MORP_DISABLED = 1 # queue is off
+    MORP_DISABLED = 0 # queue is on
+
+
+### MORP_PREFIX
+
+If you would like to have your queue names prefixed with something (eg, `prod` or `dev`) then you can set this environment variable and it will be prefixed to the queue name.
+
+
+### MORP_DSN
+
+Set this environment variable with your connection DSN so Morp can automatically configure itself when the interface is first requested.
+
+
+## FAQ
+
+### I would like to have multiple queues
+
+By default, Morp will send any message from any `morp.Message` derived class to `Message.get_name()`, you can override this behavior by giving your child class a `._name` property:
+
+```python
+from morp import Message
+
+class childMsg(Message):
+    _name = "custom-queue-name"
+```

{morp-5.2.1 → morp-7.0.0}/morp/__init__.py

@@ -19,5 +19,5 @@ from .exception import (
 )
 
 
-__version__ = '5.2.1'
+__version__ = "7.0.0"
 

{morp-5.2.1 → morp-7.0.0}/morp/config.py

@@ -1,11 +1,8 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals, division, print_function, absolute_import
-import hashlib
-import os
 import base64
 
 import dsnparse
-from datatypes import ReflectClass
+from datatypes import ReflectName
 from datatypes import property as cachedproperty
 from datatypes.config import Environ
 
@@ -14,9 +11,11 @@ from .compat import *
 
 
 class Connection(object):
-    """The base connection class, you will most likely always use DsnConnection"""
+    """The base connection class, you will most likely always use DsnConnection
+    """
     name = ""
-    """str, the name of this connection, handy when you have more than one used interface (eg, nsq)"""
+    """str, the name of this connection, handy when you have more than one used
+    interface (eg, nsq)"""
 
     username = ""
     """str, the username (if needed)"""
@@ -31,14 +30,15 @@ class Connection(object):
     """str, the path (if applicable)"""
 
     interface_name = ""
-    """str, full Interface class name -- the interface that will connect to the messaging backend"""
+    """str, full Interface class name -- the interface that will connect to the
+    messaging backend"""
 
     options = None
     """dict, any other interface specific options you need"""
 
     @property
     def interface_class(self):
-        interface_class = ReflectClass.get_class(self.interface_name)
+        interface_class = ReflectName(self.interface_name).get_class()
         return interface_class
 
     @property
@@ -65,10 +65,10 @@ class Connection(object):
         return key
 
     def __init__(self, **kwargs):
-        """
-        set all the values by passing them into this constructor, any unrecognized kwargs get put into .options
+        """set all the values by passing them into this constructor, any
+        unrecognized kwargs get put into .options
 
-        :Example:
+        :example:
             c = Connection(
                 interface_name="...",
                 hosts=[("host", port), ("host2", port2)],
@@ -84,14 +84,23 @@ class Connection(object):
         for key, val in kwargs.items():
             if hasattr(self, key):
                 setattr(self, key, val)
+
             else:
                 self.options[key] = val
 
-        self.options.setdefault('max_timeout', 3600) # 1 hour to process message
-        self.options.setdefault('backoff_multiplier', 5) # failure backoff multiplier
+        # 1 hour to process message
+        self.options.setdefault('max_timeout', 3600)
+
+        # failure backoff multiplier
+        self.options.setdefault('backoff_multiplier', 5)
 
     def get_netlocs(self, default_port):
-        return ["{}:{}".format(h[0], default_port if h[1] is None else h[1]) for h in self.hosts]
+        return [
+            "{}:{}".format(
+                h[0],
+                default_port if h[1] is None else h[1]
+            ) for h in self.hosts
+        ]
 
     def get_option(self, key, default_val):
         return getattr(self.options, key, default_val)
@@ -101,7 +110,7 @@ class DsnConnection(Connection):
     """
     Create a connection object from a dsn in the form
 
-        InterfaceName://username:password@host:port?opt1=val1&opt2=val2#connection_name
+        InterfaceName://username:password@host:port?opt1=val1#connection_name
 
     example -- connect to amazon SQS
 
@@ -116,7 +125,7 @@ class DsnConnection(Connection):
         d = {'options': {}, 'hosts': []}
         parser = dsnparse.parse(dsn)
         p = parser.fields
-        p["dsn"] = parser.parser.dsn
+        d["dsn"] = parser.parser.dsn
 
         # get the scheme, which is actually our interface_name
         d['interface_name'] = self.normalize_scheme(p["scheme"])
@@ -137,7 +146,12 @@ class DsnConnection(Connection):
         ret = v
         d = {
             "morp.interface.sqs:SQS": set(["sqs"]),
-            "morp.interface.dropfile:Dropfile": set(["dropfile"]),
+            "morp.interface.dropfile:Dropfile": set(["dropfile", "file"]),
+            "morp.interface.postgres:Postgres": set([
+                "postgres",
+                "postgresql",
+                "psql",
+            ]),
         }
 
         kv = v.lower()

{morp-5.2.1 → morp-7.0.0}/morp/interface/__init__.py

@@ -1,20 +1,9 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals, division, print_function, absolute_import
-import logging
-import sys
-from contextlib import contextmanager
-import base64
-import datetime
-
-from cryptography.fernet import Fernet
+
 import dsnparse
 
 from ..compat import *
 from ..config import DsnConnection
-from ..exception import InterfaceError
-
-
-logger = logging.getLogger(__name__)
 
 
 interfaces = {}
@@ -47,20 +36,23 @@ def find_environ(dsn_env_name='MORP_DSN', connection_class=DsnConnection):
 
     configure interfaces based on environment variables
 
-    by default, when morp is imported, it will look for MORP_DSN, and MORP_DSN_N (where
-    N is 1 through infinity) in the environment, if it finds them, it will assume they
-    are dsn urls that morp understands and will configure connections with them. If you
-    don't want this behavior (ie, you want to configure morp manually) then just make sure
-    you don't have any environment variables with matching names
-
-    The num checks (eg MORP_DSN_1, MORP_DSN_2) go in order, so you can't do MORP_DSN_1, MORP_DSN_3,
-    because it will fail on _2 and move on, so make sure your num dsns are in order (eg, 1, 2, 3, ...)
-
-    :param dsn_env_name: str, the environment variable name to find the DSN, this
-        will aslo check for values of <dsn_env_name>_N where N is 1 -> N, so you
-        can configure multiple DSNs in the environment and this will pick them all
-        up
-    :param connection_class: Connection, the class that will receive the dsn values
+    by default, when morp is imported, it will look for MORP_DSN, and MORP_DSN_N
+    (where N is 1 through infinity) in the environment, if it finds them, it
+    will assume they are dsn urls that morp understands and will configure
+    connections with them. If you don't want this behavior (ie, you want to
+    configure morp manually) then just make sure you don't have any environment
+    variables with matching names
+
+    The num checks (eg MORP_DSN_1, MORP_DSN_2) go in order, so you can't do
+    MORP_DSN_1, MORP_DSN_3, because it will fail on _2 and move on, so make sure
+    your num dsns are in order (eg, 1, 2, 3, ...)
+
+    :param dsn_env_name: str, the environment variable name to find the DSN,
+        this will aslo check for values of <dsn_env_name>_N where N is 1 -> N,
+        so you can configure multiple DSNs in the environment and this will pick
+        them all up
+    :param connection_class: Connection, the class that will receive the dsn
+        values
     :returns: generator<connection_class>
     """
     return dsnparse.parse_environs(dsn_env_name, parse_class=connection_class)
@@ -77,10 +69,11 @@ def configure(dsn, connection_class=DsnConnection):
     """
     configure an interface to be used to send messages to a backend
 
-    you use this function to configure an Interface using a dsn, then you can get
-    that interface using the get_interface() method
+    you use this function to configure an Interface using a dsn, then you can
+    get that interface using the get_interface() method
 
-    :param dsn: string, a properly formatted morp dsn, see DsnConnection for how to format the dsn
+    :param dsn: string, a properly formatted morp dsn, see DsnConnection for how
+        to format the dsn
     """
     #global interfaces
     c = dsnparse.parse(dsn, parse_class=connection_class)