harp-proxy 0.5.0b3__tar.gz

harp_proxy-0.5.0b3/LICENSE.rst

@@ -0,0 +1,43 @@
+License
+=======
+
+HARP Community Edition (or HARP CE) is the open-source core (`this repository <https://github.com/msqd/harp>`_) for the
+HARP ecosystem (including both the open-source core, the professional and enterprise extensions and third party
+extensions that did not define their own licence).
+
+
+HARP Community Edition (HARP CE)
+::::::::::::::::::::::::::::::::
+
+HARP CE is an API Runtime Proxy, a sidecar service for web applications that helps raising reliability, performances,
+security and observability of your external API usage.
+
+All the code included in this repository is part of HARP CE, and is distributed under the terms of the GNU Lesser
+General Public License (LGPL) version 3.0 or later.
+
+Copyright (C) 2023-2024  Romain Dorgueil
+
+.. code-block:: text
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Lesser General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Read the full license: https://www.gnu.org/licenses/lgpl-3.0.txt
+
+
+HARP Professional Edition (HARP PE) and Enterprise Edition (HARP EE)
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+
+HARP Professional Edition (HARP PE) and Enterprise Edition (HARP EE) are the commercial extensions of HARP CE, and are
+distributed under a commercial license. They are not open-source (although you will get access to the source code if
+you are a professional edition or enterprise edition customer), and are not distributed with this repository.

harp_proxy-0.5.0b3/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.1
+Name: harp-proxy
+Version: 0.5.0b3
+Summary: 
+Author: Romain Dorgueil
+Author-email: romain@makersquad.fr
+Requires-Python: >=3.11,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: dev
+Requires-Dist: aiohttp (>=3.9.3,<4.0.0)
+Requires-Dist: aiosqlite (>=0.19,<0.21)
+Requires-Dist: anyio (>=4.3.0,<5.0.0)
+Requires-Dist: asgimiddlewarestaticfile (>=0.6.0,<0.7.0)
+Requires-Dist: asgiref (>=3.8.1,<4.0.0)
+Requires-Dist: asyncpg (>=0.29.0,<0.30.0)
+Requires-Dist: dataclasses-json (>=0.6.3,<0.7.0)
+Requires-Dist: dataclasses-jsonschema (>=2.16.0,<3.0.0)
+Requires-Dist: essentials-configuration (>=2.0.4,<3.0.0)
+Requires-Dist: hishel (>=0.0.21,<0.0.26)
+Requires-Dist: honcho (>=1.1.0,<2.0.0) ; extra == "dev"
+Requires-Dist: http-router (>=4.1.2,<5.0.0)
+Requires-Dist: httpx (>=0.26,<0.28)
+Requires-Dist: hypercorn (>=0.16.0,<0.17.0)
+Requires-Dist: multidict (>=6.0.5,<7.0.0)
+Requires-Dist: orjson (>=3.10.0,<4.0.0)
+Requires-Dist: passlib (>=1.7.4,<2.0.0)
+Requires-Dist: psycopg2-binary (>=2.9.9,<3.0.0)
+Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
+Requires-Dist: rich (>=13.7.1,<14.0.0)
+Requires-Dist: rodi (>=2.0.6,<3.0.0)
+Requires-Dist: sentry-sdk (>=1.44.0,<2.0.0)
+Requires-Dist: sqlalchemy-utils (>=0.41.1,<0.42.0)
+Requires-Dist: sqlalchemy[asyncio] (>=2.0.29,<3.0.0)
+Requires-Dist: structlog (>=24.1.0,<25.0.0)
+Requires-Dist: svix-ksuid (>=0.6.2,<0.7.0)
+Requires-Dist: watchfiles (>=0.21.0,<0.22.0) ; extra == "dev"
+Requires-Dist: whistle (==2.0.0a1)
+Description-Content-Type: text/x-rst
+
+HARP – Harp, an API Runtime Proxy
+=================================
+
+HARP is a powerful sidecar proxy service designed to elevate the reliability, performance, security, and observability
+of your application's external API interactions. Think of it as a reverse API gateway or a nearline service mesh for
+external APIs.
+
+
+.. important::
+
+    HARP is released as an Early Access Preview.
+
+    Although we already use it for production workloads, it is still under heavy active development and some features
+    may not be available, or it may not be suitable for your applications.
+
+    We are actively looking for feedback, so please don't hesitate to reach out to us with your thoughts, ideas, or
+    issues. We are here to help you and make HARP better for everyone.
+
+
+**Quick links:** `Documentation <https://msqd.github.io/harp/>`_ | `Repository <https://github.com/msqd/harp>`_ | `Issues <https://github.com/msqd/harp/issues>`_ | `CI/CD <https://gitlab.com/makersquad/oss/harp/-/pipelines>`_
+
+
+How it works?
+:::::::::::::
+
+Overview
+--------
+
+.. image:: docs/images/HowItWorks-Overview.png
+    :alt: An overview of how HARP works in your system
+    :align: center
+
+* **Easy Integration:** Integrating with your application is a breeze, because it speaks the same language you already
+  use: HTTP. Simply run the proxy, update your API endpoints, and you're ready to go. No complex setup or extensive
+  configuration required. Of course, everything is modular and configurable, so you'll be able to fine-tune for your
+  taste later.
+* **Reduced Network Distance:** As a *nearline* proxy, HARP minimizes the network distance between your application
+  and external services, when possible, using standard techniques like caching or prefetching. This leads to faster
+  response times and improved overall performance of your application, even before any configuration is done.
+* **Simplified Development:** HARP eliminates the need for writing extensive code for functionalities such as caching,
+  monitoring, alerting, rate limiting, circuit breaking, retries, tracing, logging, and more. These features can be
+  easily delegated to the proxy, reducing the amount of code you need to write and maintain.
+* **Seamless HTTP Integration:** HARP seamlessly integrates with your application using the HTTP protocol. This means
+  that the integration or removal cost is virtually zero, or even negative when considering the reduction in code you
+  won't need to write.
+
+`Read the full documentation <https://msqd.github.io/harp/>`_
+
+
+Service
+-------
+
+.. image:: docs/images/HowItWorks-Service.png
+    :alt: What happens within the harp service
+    :align: center
+
+Within the service, harp runs one or more proxies, each one listening to one port to instrument your external API calls
+with the features you need (auditing, caching, alerting, circuit breaker switch, health checks, etc.).
+
+An additional (optional) port serves a dashboard to observe your proxies in real-time.
+
+
+Proxy
+-----
+
+.. image:: docs/images/HowItWorks-Proxy.png
+    :alt: What happens within one harp proxy
+    :align: center
+
+Each proxy is configured to intercept and forward requests to a specific external API, with independent configuration.
+
+The instance's storage is used by each proxy to store whatever it needs, depending on the activated features.
+
+Proxy features
+--------------
+
+
+Quick start
+:::::::::::
+
+Install
+
+TODO: fix once released
+
+Run
+
+TODO: fix once released
+
+Read the docs
+
+TODO: fix once released
+
+Tell us what you think
+
+TODO: fix once released
+
+
+Credits
+:::::::
+
+* Core contributors:
+
+  - `Arthur Degonde <https://github.com/ArthurD1>`_
+  - `Romain Dorgueil <https://github.com/hartym>`_
+
+* Original idea, design, and development:
+
+  - `Romain Dorgueil <https://github.com/hartym>`_
+
+* Sponsored by `Makersquad <https://www.makersquad.fr/>`_
+
+`There are many ways you can contribute to HARP! <https://msqd.github.io/harp/development/index.html>`_
+

harp_proxy-0.5.0b3/README.rst

@@ -0,0 +1,112 @@
+HARP – Harp, an API Runtime Proxy
+=================================
+
+HARP is a powerful sidecar proxy service designed to elevate the reliability, performance, security, and observability
+of your application's external API interactions. Think of it as a reverse API gateway or a nearline service mesh for
+external APIs.
+
+
+.. important::
+
+    HARP is released as an Early Access Preview.
+
+    Although we already use it for production workloads, it is still under heavy active development and some features
+    may not be available, or it may not be suitable for your applications.
+
+    We are actively looking for feedback, so please don't hesitate to reach out to us with your thoughts, ideas, or
+    issues. We are here to help you and make HARP better for everyone.
+
+
+**Quick links:** `Documentation <https://msqd.github.io/harp/>`_ | `Repository <https://github.com/msqd/harp>`_ | `Issues <https://github.com/msqd/harp/issues>`_ | `CI/CD <https://gitlab.com/makersquad/oss/harp/-/pipelines>`_
+
+
+How it works?
+:::::::::::::
+
+Overview
+--------
+
+.. image:: docs/images/HowItWorks-Overview.png
+    :alt: An overview of how HARP works in your system
+    :align: center
+
+* **Easy Integration:** Integrating with your application is a breeze, because it speaks the same language you already
+  use: HTTP. Simply run the proxy, update your API endpoints, and you're ready to go. No complex setup or extensive
+  configuration required. Of course, everything is modular and configurable, so you'll be able to fine-tune for your
+  taste later.
+* **Reduced Network Distance:** As a *nearline* proxy, HARP minimizes the network distance between your application
+  and external services, when possible, using standard techniques like caching or prefetching. This leads to faster
+  response times and improved overall performance of your application, even before any configuration is done.
+* **Simplified Development:** HARP eliminates the need for writing extensive code for functionalities such as caching,
+  monitoring, alerting, rate limiting, circuit breaking, retries, tracing, logging, and more. These features can be
+  easily delegated to the proxy, reducing the amount of code you need to write and maintain.
+* **Seamless HTTP Integration:** HARP seamlessly integrates with your application using the HTTP protocol. This means
+  that the integration or removal cost is virtually zero, or even negative when considering the reduction in code you
+  won't need to write.
+
+`Read the full documentation <https://msqd.github.io/harp/>`_
+
+
+Service
+-------
+
+.. image:: docs/images/HowItWorks-Service.png
+    :alt: What happens within the harp service
+    :align: center
+
+Within the service, harp runs one or more proxies, each one listening to one port to instrument your external API calls
+with the features you need (auditing, caching, alerting, circuit breaker switch, health checks, etc.).
+
+An additional (optional) port serves a dashboard to observe your proxies in real-time.
+
+
+Proxy
+-----
+
+.. image:: docs/images/HowItWorks-Proxy.png
+    :alt: What happens within one harp proxy
+    :align: center
+
+Each proxy is configured to intercept and forward requests to a specific external API, with independent configuration.
+
+The instance's storage is used by each proxy to store whatever it needs, depending on the activated features.
+
+Proxy features
+--------------
+
+
+Quick start
+:::::::::::
+
+Install
+
+TODO: fix once released
+
+Run
+
+TODO: fix once released
+
+Read the docs
+
+TODO: fix once released
+
+Tell us what you think
+
+TODO: fix once released
+
+
+Credits
+:::::::
+
+* Core contributors:
+
+  - `Arthur Degonde <https://github.com/ArthurD1>`_
+  - `Romain Dorgueil <https://github.com/hartym>`_
+
+* Original idea, design, and development:
+
+  - `Romain Dorgueil <https://github.com/hartym>`_
+
+* Sponsored by `Makersquad <https://www.makersquad.fr/>`_
+
+`There are many ways you can contribute to HARP! <https://msqd.github.io/harp/development/index.html>`_

harp_proxy-0.5.0b3/harp/__init__.py

@@ -0,0 +1,101 @@
+"""
+The Core (:mod:`harp`) package is the root namespace of the Harp framework.
+
+It mostly contains a reference to the :class:`Config` class, because it's the only object you need to start using Harp
+using the python API (you don't *need* to use this API, configuration files should be enough for most use cases, but
+if you want to, this is the starting point).
+
+For convenience, the :func:`run` function is also available, which is a simple way to start the default server
+implementation for your configuration object.
+
+Example usage:
+
+.. code-block:: python
+
+    from harp import Config, run
+
+    config = Config()
+    config.add_defaults()
+
+    if __name__ == "__main__":
+        run(config)
+
+You can find more information about how configuration works in the :mod:`harp.config` module.
+
+Contents
+--------
+
+"""
+
+import os
+from subprocess import check_output
+
+from packaging.version import InvalidVersion, Version
+
+ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _parse_version(version: str, /, *, default=None) -> Version:
+    try:
+        return Version(version)
+    except InvalidVersion:
+        if "-" in version:
+            return _parse_version(version.rsplit("-", 1)[0], default=default)
+        return default
+
+
+# last release
+__title__ = "Core"
+__version__ = "0.5.0b3"
+__revision__ = __version__  # we can't commit the not yet known revision
+
+# override with version.txt if available (after docker build for example)
+if os.path.exists(os.path.join(ROOT_DIR, "version.txt")):
+    with open(os.path.join(ROOT_DIR, "version.txt")) as f:
+        __version__ = f.read().strip()
+
+__parsed_version__ = _parse_version(__version__)
+
+# override with current development version/revision if available (disabled in CI, for docs)
+if not os.environ.get("CI", False) and os.path.exists(os.path.join(ROOT_DIR, ".git")):
+    __revision__ = check_output(["git", "rev-parse", "HEAD"], cwd=ROOT_DIR).decode("utf-8").strip()
+    try:
+        __version__ = (
+            check_output(["git", "describe", "--tags", "--always", "--dirty"], cwd=ROOT_DIR).decode("utf-8").strip()
+        )
+        __parsed_version__ = _parse_version(__version__, default=__parsed_version__)
+    except Exception:
+        __version__ = __revision__[:7]
+
+from ._logging import get_logger  # noqa: E402, isort: skip
+from harp.config import Config  # noqa: E402, isort: skip
+
+
+def run(config: Config):
+    """
+    Run the default server using provided configuration.
+
+    :param config: Config
+    :return:
+    """
+    import asyncio
+    import sys
+
+    from harp.config.adapters.hypercorn import HypercornAdapter
+    from harp.config.factories.kernel_factory import KernelFactory
+
+    config.read_env(sys.argv[1:])
+    factory = KernelFactory(config)
+    server = HypercornAdapter(factory)
+    asyncio.run(server.serve())
+
+
+__all__ = [
+    "Config",
+    "ROOT_DIR",
+    "__revision__",
+    "__version__",
+    "__parsed_version__",
+    "get_logger",
+    "run",
+]

harp_proxy-0.5.0b3/harp/__main__.py

@@ -0,0 +1,4 @@
+if __name__ == "__main__":
+    from .commandline import entrypoint
+
+    entrypoint()

harp_proxy-0.5.0b3/harp/_logging.py

@@ -0,0 +1,87 @@
+import logging.config
+import os
+from typing import Any
+
+import structlog
+
+processors = [
+    structlog.stdlib.filter_by_level,
+    structlog.contextvars.merge_contextvars,
+    structlog.stdlib.add_logger_name,
+    structlog.stdlib.add_log_level,
+    structlog.processors.TimeStamper(fmt="iso"),
+    structlog.dev.set_exc_info,
+    structlog.stdlib.PositionalArgumentsFormatter(),
+    structlog.processors.StackInfoRenderer(),
+    structlog.processors.UnicodeDecoder(),
+    structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
+]
+
+structlog.configure(
+    logger_factory=structlog.stdlib.LoggerFactory(),
+    wrapper_class=structlog.stdlib.BoundLogger,
+    processors=processors,
+    cache_logger_on_first_use=True,
+)
+
+LOGGING_FORMATTERS = {
+    "json": {
+        "()": structlog.stdlib.ProcessorFormatter,
+        "processor": structlog.processors.JSONRenderer(),
+    },
+    "plain": {
+        "()": structlog.stdlib.ProcessorFormatter,
+        "processor": structlog.dev.ConsoleRenderer(exception_formatter=structlog.dev.plain_traceback, colors=False),
+    },
+    "pretty": {
+        "()": structlog.stdlib.ProcessorFormatter,
+        "processor": structlog.dev.ConsoleRenderer(exception_formatter=structlog.dev.plain_traceback),
+    },
+    "keyvalue": {
+        "()": structlog.stdlib.ProcessorFormatter,
+        "processor": structlog.processors.KeyValueRenderer(key_order=["timestamp", "level", "event", "logger"]),
+    },
+}
+
+LOGGING_DEFAULT_FORMATTER = "pretty"
+LOGGING_FORMATTER = os.environ.get("LOGGING_FORMATTER", LOGGING_DEFAULT_FORMATTER)
+if LOGGING_FORMATTER not in LOGGING_FORMATTERS:
+    LOGGING_FORMATTER = LOGGING_DEFAULT_FORMATTER
+
+logging_config = {
+    "version": 1,
+    "disable_existing_loggers": True,
+    "formatters": LOGGING_FORMATTERS,
+    "handlers": {
+        "console": {
+            "class": "logging.StreamHandler",
+            "formatter": LOGGING_FORMATTER,
+        },
+    },
+    "root": {
+        "handlers": ["console"],
+        "level": logging.INFO,
+    },
+    "loggers": {
+        "harp": {"level": os.environ.get("LOGGING_HARP", "INFO")},
+        "harp.event_dispatcher": {"level": os.environ.get("LOGGING_HARP_EVENTS", "WARNING")},
+        "httpcore": {"level": os.environ.get("LOGGING_HTTPCORE", "INFO")},  # todo wrap in structlog
+        "httpx": {"level": os.environ.get("LOGGING_HTTPX", "WARNING")},  # todo wrap in structlog
+        "hypercorn.access": {"level": os.environ.get("LOGGING_HYPERCORN_ACCESS", "WARNING")},
+        "hypercorn.error": {"level": os.environ.get("LOGGING_HYPERCORN_ERROR", "INFO")},
+    },
+}
+
+
+logging.config.dictConfig(logging_config)
+
+
+def get_logger(name, *args: Any, **initial_values: Any) -> Any:
+    pkg, mod = name.rsplit(".", 1)
+    if mod in ("__init__", "__main__", "__app__"):
+        name = pkg
+
+    return structlog.get_logger(name, *args, **initial_values)
+
+
+__all__ = ["get_logger"]

harp_proxy-0.5.0b3/harp/asgi/__init__.py

@@ -0,0 +1,7 @@
+from .kernel import ASGIKernel
+
+__title__ = "ASGI"
+
+__all__ = [
+    "ASGIKernel",
+]

harp_proxy-0.5.0b3/harp/asgi/bridge/requests.py

@@ -0,0 +1,71 @@
+from typing import AsyncIterator, Optional, cast
+from urllib.parse import parse_qsl
+
+from asgiref.typing import ASGIReceiveCallable, HTTPScope
+from multidict import CIMultiDict, MultiDict
+
+from harp.http import HttpRequestBridge
+
+_default_host = (None, None)
+
+
+class HttpRequestAsgiBridge(HttpRequestBridge):
+    """Actually implements the getters required by HttpRequest using the asgi scope and receive callable. It is still
+    an early implementation and will need to support streaming requests in the future."""
+
+    def __init__(self, scope: HTTPScope, receive: ASGIReceiveCallable):
+        """
+        :param scope: see https://asgi.readthedocs.io/en/latest/specs/www.html#http-connection-scope
+        :param receive:
+        """
+        self.asgi_scope = scope
+        self.asgi_receive = receive
+
+        self._closed = False
+
+    def get_server_ipaddr(self) -> Optional[str]:
+        """Get the server IP address from asgi scope."""
+        try:
+            return (self.asgi_scope.get("server", _default_host) or _default_host)[0]
+        except IndexError:
+            return None
+
+    def get_server_port(self) -> Optional[int]:
+        """Get the server port from asgi scope."""
+        try:
+            return (self.asgi_scope.get("server", _default_host) or _default_host)[1]
+        except IndexError:
+            return None
+
+    def get_method(self) -> str:
+        """Get the HTTP method from asgi scope."""
+        return self.asgi_scope.get("method", None)
+
+    def get_path(self) -> str:
+        """Get the HTTP path from asgi scope (/foo/bar), without the query string part."""
+        if "raw_path" in self.asgi_scope:
+            return self.asgi_scope["raw_path"].decode("utf-8")
+        return self.asgi_scope.get("path", "/")
+
+    def get_query(self) -> MultiDict:
+        """Get the query string from asgi scope, as a multidict."""
+        return MultiDict(parse_qsl(self.asgi_scope.get("query_string", b"").decode("utf-8"), keep_blank_values=True))
+
+    def get_headers(self) -> CIMultiDict:
+        """Get the headers from asgi scope, as a case-insensitive multidict."""
+        headers = CIMultiDict()
+        for name, value in self.asgi_scope.get("headers", []):
+            headers.add(name.decode("utf-8"), value.decode("utf-8"))
+        return headers
+
+    async def stream(self) -> AsyncIterator[bytes]:
+        """Returns an async iterator that reads the request body chunks. This can only be used once, as it delegates to
+        the ASGI receive callable."""
+        if self._closed:
+            raise RuntimeError("Request body has already been read.")
+
+        # TODO max iter  to avoid infinite loop on erroneous proto content?
+        while not self._closed:
+            message = await self.asgi_receive()
+            self._closed = not message.get("more_body", False)
+            yield cast(bytes, message.get("body", b""))

harp_proxy-0.5.0b3/harp/asgi/bridge/responses.py

@@ -0,0 +1,42 @@
+from typing import TYPE_CHECKING
+
+from asgiref.typing import ASGISendCallable
+from multidict import CIMultiDictProxy
+
+from harp.utils.bytes import ensure_bytes
+
+if TYPE_CHECKING:
+    from harp.http import HttpResponse
+
+
+class HttpResponseAsgiBridge:  # todo protocol HttpResponseBridge
+    """Implements the ability of sending our HttpResponse object over the asgi protocol. This is still early impl. and
+    it will need to support streaming responses in the future."""
+
+    def __init__(self, response: "HttpResponse", send: ASGISendCallable):
+        self.response = response
+        self.asgi_send = send
+
+    async def send(self):
+        # set the headers as read only
+        self.response._headers = CIMultiDictProxy(self.response._headers)
+
+        # prepare headers for asgi
+        headers = tuple((ensure_bytes(k), ensure_bytes(v)) for k, v in self.response.headers.items())
+
+        # start the response
+        await self.asgi_send(
+            {
+                "type": "http.response.start",
+                "status": self.response.status,
+                "headers": headers,
+            }
+        )
+
+        # send the body
+        await self.asgi_send(
+            {
+                "type": "http.response.body",
+                "body": ensure_bytes(self.response.body),
+            }
+        )