callite 0.1.4__tar.gz

callite-0.1.4/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.1
+Name: callite
+Version: 0.1.4
+Summary: Slim Redis RPC implementation
+License: MIT
+Author: Emrah Gozcu
+Author-email: gozcu@gri.ai
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Requires-Dist: redis (>=5.0.3,<6.0.0)
+Description-Content-Type: text/markdown
+
+# RPClite
+
+RPClite is a lightweight Remote Procedure Call (RPC) implementation over Redis, designed to facilitate communication between different components of a distributed system. It minimizes dependencies and offers a simple yet effective solution for decoupling complex systems, thus alleviating potential library conflicts.
+
+## Setting up RPClite
+
+Before using RPClite, ensure you have a Redis instance running. You can start a Redis server using the default settings or configure it as per your requirements.
+
+## Implementing the Server
+
+To implement the RPClite server, follow these steps:
+
+1. Import the `RPCService` class from `server.rpc_server`.
+2. Define your main class and initialize the RPC service with the Redis URL and service name.
+3. Register your functions with the RPC service using the `register` decorator.
+4. Run the RPC service indefinitely.
+
+Here's an example implementation:
+
+```python
+from callite.server import RPCService
+
+
+class Main:
+    def __init__(self):
+        self.service = "service"
+        self.redis_url = "redis://redis:6379/0"
+        self.rpc_service = RPCService(self.redis_url, self.service)
+
+    def run(self):
+        @self.rpc_service.register
+        def healthcheck():
+            return "OK"
+
+        self.rpc_service.run_forever()
+
+
+if __name__ == "__main__":
+    Main().run()
+```
+
+## Calling the Function from Client
+
+Once the server is set up, you can call functions remotely from the client side. Follow these steps to call functions:
+
+1. Import the `RPCClient` class from `client.rpc_client`.
+2. Define your client class and initialize the RPC client with the Redis URL and service name.
+3. Call the function using the `execute` method of the RPC client.
+4. Optionally, you can pass arguments and keyword arguments to the function.
+
+Here's an example client implementation:
+
+```python
+import time
+from callite.client.rpc_client import RPCClient
+
+
+class Healthcheck():
+    def __init__(self):
+        self.status = "OK"
+        self.r = RPCClient("redis://redis:6379/0", "service")
+
+    def get_status(self):
+        start = time.perf_counter()
+        self.status = self.r.execute('healthcheck')
+        end = time.perf_counter()
+        print(f"Healthcheck took {end - start:0.4f} seconds")
+        return self.status
+
+    def check(self):
+        return self.get_status()
+
+
+if __name__ == "__main__":
+    Healthcheck().check()
+```
+
+You can pass arguments and keyword arguments to the `execute` method as follows:
+
+```python
+self.status = self.r.execute('healthcheck', [True], {'a': 1, 'b': 2})
+```
+
+This setup allows for efficient communication between components of a distributed system, promoting modularity and scalability.
+```

callite-0.1.4/README.md

@@ -0,0 +1,85 @@
+# RPClite
+
+RPClite is a lightweight Remote Procedure Call (RPC) implementation over Redis, designed to facilitate communication between different components of a distributed system. It minimizes dependencies and offers a simple yet effective solution for decoupling complex systems, thus alleviating potential library conflicts.
+
+## Setting up RPClite
+
+Before using RPClite, ensure you have a Redis instance running. You can start a Redis server using the default settings or configure it as per your requirements.
+
+## Implementing the Server
+
+To implement the RPClite server, follow these steps:
+
+1. Import the `RPCService` class from `server.rpc_server`.
+2. Define your main class and initialize the RPC service with the Redis URL and service name.
+3. Register your functions with the RPC service using the `register` decorator.
+4. Run the RPC service indefinitely.
+
+Here's an example implementation:
+
+```python
+from callite.server import RPCService
+
+
+class Main:
+    def __init__(self):
+        self.service = "service"
+        self.redis_url = "redis://redis:6379/0"
+        self.rpc_service = RPCService(self.redis_url, self.service)
+
+    def run(self):
+        @self.rpc_service.register
+        def healthcheck():
+            return "OK"
+
+        self.rpc_service.run_forever()
+
+
+if __name__ == "__main__":
+    Main().run()
+```
+
+## Calling the Function from Client
+
+Once the server is set up, you can call functions remotely from the client side. Follow these steps to call functions:
+
+1. Import the `RPCClient` class from `client.rpc_client`.
+2. Define your client class and initialize the RPC client with the Redis URL and service name.
+3. Call the function using the `execute` method of the RPC client.
+4. Optionally, you can pass arguments and keyword arguments to the function.
+
+Here's an example client implementation:
+
+```python
+import time
+from callite.client.rpc_client import RPCClient
+
+
+class Healthcheck():
+    def __init__(self):
+        self.status = "OK"
+        self.r = RPCClient("redis://redis:6379/0", "service")
+
+    def get_status(self):
+        start = time.perf_counter()
+        self.status = self.r.execute('healthcheck')
+        end = time.perf_counter()
+        print(f"Healthcheck took {end - start:0.4f} seconds")
+        return self.status
+
+    def check(self):
+        return self.get_status()
+
+
+if __name__ == "__main__":
+    Healthcheck().check()
+```
+
+You can pass arguments and keyword arguments to the `execute` method as follows:
+
+```python
+self.status = self.r.execute('healthcheck', [True], {'a': 1, 'b': 2})
+```
+
+This setup allows for efficient communication between components of a distributed system, promoting modularity and scalability.
+```

callite-0.1.4/callite/__init__.py

@@ -0,0 +1,5 @@
+# import all the classe
+from .client import RPCClient
+from .server import RPCServer
+from .rpctypes import MessageBase, Request, Response, RPCException
+from .shared import RedisConnection

callite-0.1.4/callite/client/__init__.py

@@ -0,0 +1,2 @@
+from .rpc_client import *
+__all__ = ['RPCClient']

callite-0.1.4/callite/client/rpc_client.py

@@ -0,0 +1,64 @@
+import json
+import threading
+import logging
+
+from callite.shared.redis_connection import RedisConnection
+from callite.rpctypes.request import Request
+
+
+# import pydevd_pycharm
+# pydevd_pycharm.settrace('host.docker.internal', port=4444, stdoutToServer=True, stderrToServer=True)
+
+
+class RPCClient(RedisConnection):
+
+    def __init__(self, conn_url: str, service: str, *args, **kwargs) -> None:
+        super().__init__(conn_url, service, *args, **kwargs)
+        self._request_pool = {}
+        self._subscribe()
+        self._logger = logging.getLogger(__name__)
+        self._logger.addHandler(logging.StreamHandler())
+        self._logger.setLevel(logging.INFO)
+
+    def _subscribe(self):
+        self.channel = self._rds.pubsub()
+        self.channel.subscribe(f'{self._queue_prefix}/response/{self._connection_id}')
+        thread = threading.Thread(target=self._pull_from_redis, daemon=True)
+        thread.start()
+
+    def _pull_from_redis(self):
+        def _on_delivery(message):
+            self._logger.info(f"Received message {message}")
+            data = json.loads(message['data'].decode('utf-8'))
+            request_guid = data['request_id']
+            if request_guid not in self._request_pool:
+                return
+            lock, _ = self._request_pool.pop(request_guid)
+            self._request_pool[request_guid] = (lock, data)
+            lock.release()
+
+        # TODO: handle poisonous messages from redis (e.g. non-json, old messages, etc.)
+        while self._running:
+            message: dict | None = self.channel.get_message(ignore_subscribe_messages=True, timeout=100)
+            if not message: continue
+            if not message['data']: continue
+            _on_delivery(message)
+
+
+    def execute(self, method: str, *args, **kwargs) -> dict:
+        request = Request(method, self._connection_id, None, *args, **kwargs)
+        request_uuid = request.request_id
+
+        request_lock = threading.Lock()
+        self._request_pool[request_uuid] = (request_lock, None)
+        request_lock.acquire()
+
+        self._rds.xadd(f'{self._queue_prefix}/request/{self._service}', {'data': json.dumps(request.payload_json())})
+        self._logger.info(f"Sent message {request_uuid} to {self._queue_prefix}/request/{self._service}")
+        # TODO: parameterize timeout
+        lock_success = request_lock.acquire(timeout=30)
+        lock, response = self._request_pool.pop(request_uuid)
+        if lock_success:
+            self._logger.info(f"Received response to message {request_uuid} {type(response['data'])}")
+            return response['data']
+        raise Exception('Timeout')

callite-0.1.4/callite/rpctypes/__init__.py

@@ -0,0 +1,6 @@
+from .message_base import *
+from .request import *
+from .response import *
+from .rpc_exception import *
+
+__all__ = ['MessageBase', 'Request', 'Response', 'RPCException']

callite-0.1.4/callite/rpctypes/message_base.py

@@ -0,0 +1,7 @@
+class MessageBase(object):
+    def __init__(self, method, message_id):
+        self.message_id: str = message_id
+        self.method: str = method
+
+    def __str__(self):
+        return "MessageBase: message_id: %s, method: %s, message_data: %s %s" % (self.message_id, self.method, self.args, self.kwargs)

callite-0.1.4/callite/rpctypes/request.py

@@ -0,0 +1,25 @@
+import uuid
+
+from callite.rpctypes.message_base import MessageBase
+
+
+class Request(MessageBase):
+    def __init__(self, method: str, client_id:str, message_id = None, *args, **kwargs):
+        super(Request, self).__init__(method, message_id)
+        self.request_id = message_id if message_id else uuid.uuid4().hex
+        self.client_id = client_id
+        self.args = args
+        self.kwargs = kwargs
+
+    def set_data(self, data):
+        self.data = data
+    def payload_json(self):
+        return {
+            'request_id': self.request_id,
+            'client_id': self.client_id,
+            'method': self.method,
+            'params': {'args': self.args, 'kwargs': self.kwargs}
+        }
+
+    def __str__(self):
+        return "Request: request_id: %s, method: %s" % (self.request_id, self.method)

callite-0.1.4/callite/rpctypes/response.py

@@ -0,0 +1,18 @@
+import json
+
+from callite.rpctypes.message_base import MessageBase
+
+
+class Response(MessageBase):
+    def __init__(self, method: str, message_id = None):
+        super(Response, self).__init__(method, message_id)
+        self.data = None
+
+
+    def response_json(self):
+        # get the json string of the current instance
+        response_data = json.dumps(self).encode('utf-8')
+        return response_data
+
+    def __str__(self):
+        return "Response: message_id: %s, response_data: %s" % (self.message_id, self.data)

callite-0.1.4/callite/rpctypes/rpc_exception.py

@@ -0,0 +1,10 @@
+from callite.rpctypes.request import Request
+
+
+class RPCException(Exception):
+
+    def __init__(self, message, ex):
+        super().__init__(message)
+        self.message = message
+        self.request: Request | None = None
+        self.inner_exception = ex

callite-0.1.4/callite/server/__init__.py

@@ -0,0 +1,3 @@
+# add rpcserver to all modules
+from .rpc_server import *
+__all__ = ['RPCServer']

callite-0.1.4/callite/server/rpc_server.py

@@ -0,0 +1,89 @@
+import json
+import logging
+import threading
+import time
+from types import FunctionType
+from typing import Any, Callable
+
+import redis
+
+from callite.rpctypes.response import Response
+from callite.shared.redis_connection import RedisConnection
+
+
+# import pydevd_pycharm
+# pydevd_pycharm.settrace('host.docker.internal', port=4444, stdoutToServer=True, stderrToServer=True)
+# TODO: Check method calls and parameters
+class RPCServer(RedisConnection):
+    def __init__(self, conn_url: str, service: str, *args, **kwargs):
+        super().__init__(conn_url, service, *args, **kwargs)
+        self._registered_methods = {}
+        self._xread_groupname = kwargs.get('xread_groupname', 'generic')
+
+        t = threading.Thread(target=self._subscribe_redis, daemon=True)
+        t.start()
+        self._logger = logging.getLogger(__name__)
+        self._logger.addHandler(logging.StreamHandler())
+        self._logger.setLevel(logging.INFO)
+
+
+    def register(self, handler: FunctionType | Callable, method_name: str | None = None) -> Callable:
+        method_name = method_name or handler.__name__
+        self._registered_methods[method_name] = handler
+        return handler
+
+    def run_forever(self) -> None:
+        while self._running: time.sleep(1000000)
+
+
+    def _subscribe_redis(self):
+        self._create_redis_group()
+        while self._running:
+            messages = self._read_messages_from_redis()
+            self._process_messages(messages)
+
+    def _create_redis_group(self):
+        try:
+            self._rds.xgroup_create(f'{self._queue_prefix}/request/{self._service}', self._xread_groupname, mkstream=True)
+        except redis.exceptions.ResponseError as e:
+            if "name already exists" not in str(e): raise
+
+    def _read_messages_from_redis(self):
+        messages = self._rds.xreadgroup(self._xread_groupname, self._connection_id, {f'{self._queue_prefix}/request/{self._service}': '>'}, count=1, block=1000, noack=True)
+        self._logger.info(f"{len(messages)} messages received from {self._queue_prefix}/request/{self._service}")
+        return messages
+
+    def _process_messages(self, messages):
+        for _, message_list in messages:
+            for _message in message_list:
+                message_id, message_data = _message
+                message_data = json.loads(message_data[b'data'])
+                self._logger.info(f"Processing message {message_id} with data: {message_data}")
+                self._handle_messages(message_data, message_id)
+
+    def _handle_messages(self, message_data, message_id):
+        response = self._call_registered_method(message_data['method'], message_id, message_data['params'])
+        request_id =  message_data['request_id']
+        self._logger.info(f"Response to message {message_id} is {response}")
+        payload = json.dumps({'data': response, 'request_id': request_id})
+        self._rds.publish(f'{self._queue_prefix}/response/{message_data["client_id"]}', payload)
+        self._rds.xack(f'{self._queue_prefix}/request/{self._service}', self._xread_groupname, message_id)
+        self._logger.info(f"Processed message {message_id} and response published to {self._queue_prefix}/response/{message_data['request_id']}")
+
+
+    def _call_registered_method(self, method: str, message_id, params: dict) -> Any:
+        if method not in self._registered_methods:
+            self._logger.warn(f"Method {method} not registered")
+            return
+        try:
+            data = self._registered_methods[method](*params['args'], **params['kwargs'])
+
+            # TODO: Check why message_id is bytes
+            message_id = message_id.decode('utf-8') if isinstance(message_id, bytes) else message_id
+            response = Response(self._service, message_id)
+            response.data = data
+            return response.__dict__
+        except Exception as e:
+            self._logger.error(e)
+            # TODO: log and return exception
+            return

callite-0.1.4/callite/shared/__init__.py

@@ -0,0 +1,2 @@
+from .redis_connection import *
+__all__ = ['RedisConnection']

callite-0.1.4/callite/shared/redis_connection.py

@@ -0,0 +1,22 @@
+import logging
+import uuid
+import redis
+from abc import ABC
+
+
+
+class RedisConnection(ABC):
+    def __init__(self, conn_url: str, service: str, *args, **kwargs):
+        self._methods = {}
+        self._service = service
+        self._running = True
+        self._running_threads = []
+        self._connection_id = uuid.uuid4().hex
+        self._queue_prefix = kwargs.get('queue_prefix', '/rpclite')
+        self._rds = redis.Redis.from_url(conn_url)
+        self._rds.ping()
+        self._logger = logging.getLogger(f'RPC{service}')
+        self._logger.setLevel(logging.INFO)
+
+    def close(self) -> None:
+        self._running = False

callite-0.1.4/pyproject.toml

@@ -0,0 +1,20 @@
+[tool.poetry]
+name = "callite"
+version = "0.1.4"
+description = "Slim Redis RPC implementation"
+authors = ["Emrah Gozcu <gozcu@gri.ai>"]
+license = "MIT"
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = "^3.10"
+redis = "^5.0.3"
+
+[tool.poetry.dev-dependencies]
+mypy = "^1.9.0"
+setuptools = "^69.1.1"
+pydevd-pycharm = "~=233.14475.28"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

callite-0.1.4/setup.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+from setuptools import setup
+
+packages = \
+['callite',
+ 'callite.client',
+ 'callite.rpctypes',
+ 'callite.server',
+ 'callite.shared']
+
+package_data = \
+{'': ['*']}
+
+install_requires = \
+['redis>=5.0.3,<6.0.0']
+
+setup_kwargs = {
+    'name': 'callite',
+    'version': '0.1.4',
+    'description': 'Slim Redis RPC implementation',
+    'long_description': '# RPClite\n\nRPClite is a lightweight Remote Procedure Call (RPC) implementation over Redis, designed to facilitate communication between different components of a distributed system. It minimizes dependencies and offers a simple yet effective solution for decoupling complex systems, thus alleviating potential library conflicts.\n\n## Setting up RPClite\n\nBefore using RPClite, ensure you have a Redis instance running. You can start a Redis server using the default settings or configure it as per your requirements.\n\n## Implementing the Server\n\nTo implement the RPClite server, follow these steps:\n\n1. Import the `RPCService` class from `server.rpc_server`.\n2. Define your main class and initialize the RPC service with the Redis URL and service name.\n3. Register your functions with the RPC service using the `register` decorator.\n4. Run the RPC service indefinitely.\n\nHere\'s an example implementation:\n\n```python\nfrom callite.server import RPCService\n\n\nclass Main:\n    def __init__(self):\n        self.service = "service"\n        self.redis_url = "redis://redis:6379/0"\n        self.rpc_service = RPCService(self.redis_url, self.service)\n\n    def run(self):\n        @self.rpc_service.register\n        def healthcheck():\n            return "OK"\n\n        self.rpc_service.run_forever()\n\n\nif __name__ == "__main__":\n    Main().run()\n```\n\n## Calling the Function from Client\n\nOnce the server is set up, you can call functions remotely from the client side. Follow these steps to call functions:\n\n1. Import the `RPCClient` class from `client.rpc_client`.\n2. Define your client class and initialize the RPC client with the Redis URL and service name.\n3. Call the function using the `execute` method of the RPC client.\n4. Optionally, you can pass arguments and keyword arguments to the function.\n\nHere\'s an example client implementation:\n\n```python\nimport time\nfrom callite.client.rpc_client import RPCClient\n\n\nclass Healthcheck():\n    def __init__(self):\n        self.status = "OK"\n        self.r = RPCClient("redis://redis:6379/0", "service")\n\n    def get_status(self):\n        start = time.perf_counter()\n        self.status = self.r.execute(\'healthcheck\')\n        end = time.perf_counter()\n        print(f"Healthcheck took {end - start:0.4f} seconds")\n        return self.status\n\n    def check(self):\n        return self.get_status()\n\n\nif __name__ == "__main__":\n    Healthcheck().check()\n```\n\nYou can pass arguments and keyword arguments to the `execute` method as follows:\n\n```python\nself.status = self.r.execute(\'healthcheck\', [True], {\'a\': 1, \'b\': 2})\n```\n\nThis setup allows for efficient communication between components of a distributed system, promoting modularity and scalability.\n```',
+    'author': 'Emrah Gozcu',
+    'author_email': 'gozcu@gri.ai',
+    'maintainer': None,
+    'maintainer_email': None,
+    'url': None,
+    'packages': packages,
+    'package_data': package_data,
+    'install_requires': install_requires,
+    'python_requires': '>=3.10,<4.0',
+}
+
+
+setup(**setup_kwargs)