splight-runner 1.0.0__tar.gz

splight_runner-1.0.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.1
+Name: splight-runner
+Version: 1.0.0
+Summary: Splight Runner
+Author: Splight Dev
+Author-email: dev@splight-ae.com
+Requires-Python: >=3.8,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: requests (>=2.26.0,<3.0.0)
+Description-Content-Type: text/markdown
+
+# Splight Runner
+
+---
+
+The `splight-runner` package is a tool for running processes that may interact 
+with the **Splight Engine** in order to report monitoring information of the 
+process.
+
+## Description
+
+The `splight-runner` main functionality is to run different kinds of *Python* 
+processes related in some way to the **Splight Engine**, for example,
+*components* that were developed using the **Splight Library**. 
+
+For this reason, the different features in the runner are for reporting the 
+status of the process and centralizing logs to be accessed through the 
+**Splight Engine**. So the `splight-runner` is used when the process is 
+launched from the **Splight Engine**, this means that it should not be used 
+during the development of the component even though the process will be
+running under the `splight-runner`.
+
+## Usage
+
+The package has different commands that can be used. Here you can find a list
+of all the commands with a short description and how to use them.
+
+### Running Components
+
+The first command is used for running components that use the **Splight Library**. 
+The command is
+```bash
+splight-runner run-component <path>/main.py --component-id=<component_id>
+```
+This command will execute the component whose main file is called `main.py` and 
+will modify some default functioning in order to report the status of the component and
+send the logs of the component to the **Splight Engine**. In the next sections, you can 
+find how these modifications are done by the `splight-runner`.
+
+In order to this command work as expected some environment variables are needed. 
+The following are the list of variables that should be configured before running the 
+command:
+```bash
+SPLIGHT_ACCESS_ID=<access_id>
+SPLIGHT_SECRET_KEY=<secret_key
+SPLIGHT_PLATFORM_API_HOST=https://api.splight-ai.com
+COMPONENT_ID=<component_id>
+```
+Some of the variables are pointing to the production environment, but if you are 
+interested in using another environment like "integration" or maybe a local 
+environment you need to modify the corresponding variable value.
+
+It is important to mention that the `splight-runner` should be used for components that
+use the **Splight Lib** and **Splight CLI** grater than version `4.0.0`. For components
+with older versions we can't ensure the proper functioning.
+
+## Structure
+
+You may be asking what the `splight-runner` does in order to do whatever it does.
+Well, that's not an easy question to respond but let's try to explain a little bit.
+
+So far, the package has two main functionalities, sending logs and reporting status. 
+With the two features we already have a lot of magic is happening, but maybe in the future
+we will add some more.
+
+`splight-runner` works as a wrapper around the process that will be executed, 
+this means some configurations are done, so the process in question is executed.
+
+For example, when the command `run-component` is used, the environment variable
+`PYTHONPATH` is modified in order to include the directory where the file `sitecustomize.py`
+of the `splight-runner` is located. That file is a *Python* configuration file that can 
+be used to customize the behavior of Python's site module. The site module is 
+responsible for setting up Python's runtime environment, including configuring 
+paths, importing modules, and performing other site-specific tasks.
+
+So, using this file we can modify the default behaviors of libraries, for example, we 
+can modify the import process of modules or libraries in order to change or add
+a new behavior. This way we can modify the `logging` module and intercept 
+the logging process to show the logs messages and also send the logs messages
+to the **Splight Engine**. Some similar implementation is used for 
+reporting the component's status. 
+
+
+## Development and Contributing
+
+Since the main goal of the tool is to be used for running different kinds of 
+`Python` processes, is mandatory not to interfere with the process's 
+dependencies, this should be considered when adding new features for the 
+`splight-runner`. This is the reason why some basic functionalities 
+are implemented in the source code and not imported from a third-party 
+library, for example, a command parser for the different commands was created 
+in the code and no library was used like `click` or `typer`.
+
+The package is fully dockerized, for testing new features you can use the 
+docker image that can be build with 
+the command
+```bash
+make build
+```
+For running the docker container the command is
+```bash
+make start
+```
+and to stop the container:
+```bash
+make stop
+```
+
+It is important to note that testing new features in the `splight-runner` is 
+not an easy task, you may need to modify the docker-compose file in order to 
+include new volumes.
+

splight_runner-1.0.0/README.md

@@ -0,0 +1,110 @@
+# Splight Runner
+
+---
+
+The `splight-runner` package is a tool for running processes that may interact 
+with the **Splight Engine** in order to report monitoring information of the 
+process.
+
+## Description
+
+The `splight-runner` main functionality is to run different kinds of *Python* 
+processes related in some way to the **Splight Engine**, for example,
+*components* that were developed using the **Splight Library**. 
+
+For this reason, the different features in the runner are for reporting the 
+status of the process and centralizing logs to be accessed through the 
+**Splight Engine**. So the `splight-runner` is used when the process is 
+launched from the **Splight Engine**, this means that it should not be used 
+during the development of the component even though the process will be
+running under the `splight-runner`.
+
+## Usage
+
+The package has different commands that can be used. Here you can find a list
+of all the commands with a short description and how to use them.
+
+### Running Components
+
+The first command is used for running components that use the **Splight Library**. 
+The command is
+```bash
+splight-runner run-component <path>/main.py --component-id=<component_id>
+```
+This command will execute the component whose main file is called `main.py` and 
+will modify some default functioning in order to report the status of the component and
+send the logs of the component to the **Splight Engine**. In the next sections, you can 
+find how these modifications are done by the `splight-runner`.
+
+In order to this command work as expected some environment variables are needed. 
+The following are the list of variables that should be configured before running the 
+command:
+```bash
+SPLIGHT_ACCESS_ID=<access_id>
+SPLIGHT_SECRET_KEY=<secret_key
+SPLIGHT_PLATFORM_API_HOST=https://api.splight-ai.com
+COMPONENT_ID=<component_id>
+```
+Some of the variables are pointing to the production environment, but if you are 
+interested in using another environment like "integration" or maybe a local 
+environment you need to modify the corresponding variable value.
+
+It is important to mention that the `splight-runner` should be used for components that
+use the **Splight Lib** and **Splight CLI** grater than version `4.0.0`. For components
+with older versions we can't ensure the proper functioning.
+
+## Structure
+
+You may be asking what the `splight-runner` does in order to do whatever it does.
+Well, that's not an easy question to respond but let's try to explain a little bit.
+
+So far, the package has two main functionalities, sending logs and reporting status. 
+With the two features we already have a lot of magic is happening, but maybe in the future
+we will add some more.
+
+`splight-runner` works as a wrapper around the process that will be executed, 
+this means some configurations are done, so the process in question is executed.
+
+For example, when the command `run-component` is used, the environment variable
+`PYTHONPATH` is modified in order to include the directory where the file `sitecustomize.py`
+of the `splight-runner` is located. That file is a *Python* configuration file that can 
+be used to customize the behavior of Python's site module. The site module is 
+responsible for setting up Python's runtime environment, including configuring 
+paths, importing modules, and performing other site-specific tasks.
+
+So, using this file we can modify the default behaviors of libraries, for example, we 
+can modify the import process of modules or libraries in order to change or add
+a new behavior. This way we can modify the `logging` module and intercept 
+the logging process to show the logs messages and also send the logs messages
+to the **Splight Engine**. Some similar implementation is used for 
+reporting the component's status. 
+
+
+## Development and Contributing
+
+Since the main goal of the tool is to be used for running different kinds of 
+`Python` processes, is mandatory not to interfere with the process's 
+dependencies, this should be considered when adding new features for the 
+`splight-runner`. This is the reason why some basic functionalities 
+are implemented in the source code and not imported from a third-party 
+library, for example, a command parser for the different commands was created 
+in the code and no library was used like `click` or `typer`.
+
+The package is fully dockerized, for testing new features you can use the 
+docker image that can be build with 
+the command
+```bash
+make build
+```
+For running the docker container the command is
+```bash
+make start
+```
+and to stop the container:
+```bash
+make stop
+```
+
+It is important to note that testing new features in the `splight-runner` is 
+not an easy task, you may need to modify the docker-compose file in order to 
+include new volumes.

splight_runner-1.0.0/pyproject.toml

@@ -0,0 +1,35 @@
+[tool.poetry]
+name = "splight-runner"
+version = "1.0.0"
+description = "Splight Runner"
+authors = ["Splight Dev <dev@splight-ae.com>"]
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = "^3.8"
+requests = "^2.26.0"
+
+[tool.poetry.group.dev.dependencies]
+black = "23.3.0"
+isort = "5.12.0"
+ipdb = "^0.13.13"
+ipython = "8.12.2"
+
+[tool.poetry.scripts]
+splight-runner = "splight_runner.runner:runner"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.black]
+line-length = 79
+force-exclude = ".*_pb2.*py.*"
+
+[tool.isort]
+profile = "black"
+line_length = 79
+skip_glob = ["*_pb2*.py*"]
+
+[tool.ruff]
+line-length = 79

splight_runner-1.0.0/src/splight_runner/__init__.py

@@ -0,0 +1,3 @@
+from splight_runner.runner import runner
+
+__all__ = [runner]

splight_runner-1.0.0/src/splight_runner/api/component_reporter.py

@@ -0,0 +1,55 @@
+import requests
+
+from splight_runner.api.settings import settings
+
+
+class StatusReporter:
+    """Class responsible for updating the component status each time the
+    healthcheck method is called.
+    """
+
+    _SPL_PREFIX = "Splight"
+    _BASE_PATH = "v2/engine/component/components"
+
+    def __init__(
+        self, api_host: str, access_id: str, secret_key: str, component_id: str
+    ):
+        self._api_host = api_host.rstrip("/")
+        self._component_id = component_id
+        self._auth_header = {
+            "Authorization": f"{self._SPL_PREFIX} {access_id} {secret_key}"
+        }
+        self._prev_status: str = "Unknown"
+
+    def report_status(self, status: str) -> None:
+        """Updates the status of the component in the Splight Platform making
+        POST request to the API.
+
+        Parameters
+        ----------
+        status : str
+            The status of the component.
+            It can be "Running", "Stopped" or "Succeeded".
+        """
+        if self._prev_status == status:
+            return None
+        url = f"{self._api_host}/{self._BASE_PATH}"
+        url = f"{url}/{self._component_id}/update-status/"
+        response = requests.post(
+            url, headers=self._auth_header, data={"deployment_status": status}
+        )
+        try:
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as exc:
+            print("Unable to update component status")
+            print(exc)
+        self._prev_status = status
+        return None
+
+
+reporter = StatusReporter(
+    api_host=settings.splight_platform_api_host,
+    access_id=settings.access_id,
+    secret_key=settings.secret_key,
+    component_id=settings.process_id,
+)

splight_runner-1.0.0/src/splight_runner/api/logging_interceptor.py

@@ -0,0 +1,85 @@
+import traceback
+from logging import LogRecord
+
+from splight_runner.api.settings import settings
+from splight_runner.log_streamer.log_streamer import ComponentLogsStreamer
+from splight_runner.logging import log
+
+
+class ApplicationLogInterceptor:
+    """Class responsible for intercept logs records from the logging module
+    and append them into a queue for further processing. In particular for
+    sending the logs events to Splight GRPC API.
+    """
+
+    def __init__(
+        self,
+        host: str,
+        access_id: str,
+        secret_key: str,
+        process_id: str,
+        sender_type: str,
+    ):
+        self._streamer = ComponentLogsStreamer(
+            host=host,
+            access_id=access_id,
+            secret_key=secret_key,
+            process_id=process_id,
+        )
+        self._sender_type = sender_type
+        self._streamer.start()
+
+    def save_record(self, record: LogRecord) -> None:
+        """Saves the logging record to be sent.
+
+        Parameters
+        ----------
+        record: LogRecord
+            The log record.
+        """
+        exc_info = None
+        try:
+            message = str(record.msg) % record.args
+        except Exception as exc:
+            log(exc)
+            message = str(record.msg)
+
+        if record.exc_info:
+            exc_info = "".join(
+                traceback.format_exception(*record.exc_info)
+            ).replace('"', "'")
+            message = str(record.msg)
+        else:
+            exc_info = ""
+
+        # tags attribute is not default in logger
+        tags = None
+        if hasattr(record, "tags"):
+            tags = getattr(record, "tags")
+        event = {
+            "sender_type": self._sender_type,
+            "name": record.name,
+            "message": message,
+            "loglevel": record.levelname,
+            "filename": record.filename,
+            "traceback": exc_info,
+            "tags": tags,
+        }
+        self._streamer.insert_message(event)
+
+    def flush(self) -> None:
+        """Flushes the log records."""
+        self._streamer.flush()
+
+    def stop(self) -> None:
+        """Stops the thread."""
+        self._streamer.stop()
+
+
+interceptor = ApplicationLogInterceptor(
+    host=settings.splight_platform_api_host,
+    access_id=settings.access_id,
+    secret_key=settings.secret_key,
+    process_id=settings.process_id,
+    sender_type=settings.process_type,
+)

splight_runner-1.0.0/src/splight_runner/api/settings.py

@@ -0,0 +1,63 @@
+import os
+from dataclasses import dataclass, field
+from functools import partial
+from typing import List
+
+SECRET_DIR = "/etc/config"
+
+
+class MissingEnvVariable(Exception):
+    ...
+
+
+def load_variable(var_name: str) -> str:
+    secret_file = os.path.join(SECRET_DIR, var_name)
+    if os.path.exists(secret_file):
+        with open(secret_file) as f:
+            variable = f.read().strip()
+    elif os.getenv(var_name):
+        variable = os.getenv(var_name)
+    else:
+        raise MissingEnvVariable(f"{var_name} is missing")
+    return variable
+
+
+def load_multiple_variable(var_names: List[str]) -> str:
+    value = None
+    for var_name in var_names:
+        try:
+            value = load_variable(var_name)
+            break
+        except MissingEnvVariable:
+            continue
+
+    if value is None:
+        raise MissingEnvVariable(f"On of {var_names} is missing")
+    return value
+
+
+@dataclass
+class SplightSettings:
+    """Class for holding Splight Runner settings."""
+
+    access_id: str = field(
+        default_factory=partial(load_variable, "SPLIGHT_ACCESS_ID")
+    )
+    secret_key: str = field(
+        default_factory=partial(load_variable, "SPLIGHT_SECRET_KEY")
+    )
+    splight_platform_api_host: str = field(
+        default_factory=partial(load_variable, "SPLIGHT_PLATFORM_API_HOST")
+    )
+    process_id: str = field(
+        default_factory=partial(
+            load_multiple_variable,
+            ["COMPONENT_ID", "COMPUTE_NODE_ID", "AGENT_ID"],
+        )
+    )
+    process_type: str = field(
+        default_factory=partial(load_variable, "PROCESS_TYPE")
+    )
+
+
+settings = SplightSettings()

splight_runner-1.0.0/src/splight_runner/bootstrap/sitecustomize.py

@@ -0,0 +1,39 @@
+# This file is used for configuring splight runner in runtime. In particular
+# loads some env vars and activate hooks for sending logs.
+# All hooks or configurations for the Splight Runner should be loaded in
+# this file.
+
+import ast
+import os
+import sys
+
+runner_active = ast.literal_eval(os.getenv("SPLIGHT_RUNNER_ACTIVE", "False"))
+
+boot_directory = os.path.dirname(__file__)
+root_directory = os.path.dirname(os.path.dirname(boot_directory))
+
+path = list(sys.path)
+
+if boot_directory in path:
+    del path[path.index(boot_directory)]
+
+try:
+    from importlib.machinery import PathFinder
+
+    module_spec = PathFinder.find_spec("sitecustomize", path=path)
+except ImportError as exc:
+    sys.stdout.write("Unable to import Splight Runner sitecustomie", exc)
+    sys.stdout.flush()
+else:
+    if module_spec is not None:
+        module_spec.loader.load_module("sitecustomize")
+
+if runner_active:
+    do_insert = root_directory not in sys.path
+    if do_insert:
+        sys.path.insert(0, root_directory)
+
+    import splight_runner.config
+
+    if do_insert:
+        del sys.path[sys.path.index(root_directory)]

splight_runner-1.0.0/src/splight_runner/commands/admin.py

@@ -0,0 +1,136 @@
+import inspect
+import sys
+from typing import Any, Callable, Dict, List, Optional
+
+
+class MissingCommandError(Exception):
+    """Exception"""
+
+    ...
+
+
+class MissingArgument(Exception):
+    """Exception raises when there is a missing argument for a command"""
+
+    ...
+
+
+class CommandAdmin:
+    """Class respobsible for taking registry of all the commands and callbaks"""
+
+    def __init__(self, name: str, version: str):
+        """Constructor
+
+        Parameters
+        ----------
+        name : str
+            Name of the application
+        version : str
+            The application version
+        """
+        self._name = name
+        self._version = version
+        self._commands: Dict[str, Callable] = {}
+        self._callback: Dict[str, Callable] = {}
+
+    def command(self, name: Optional[str] = None) -> Callable:
+        """Decorator to register a command
+
+        Parameters
+        ----------
+        name: Optional[str]
+            The name for the command
+
+        Returns
+        -------
+        Callable
+            The decorated function
+        """
+
+        def decorator(func: Callable) -> Callable:
+            func_name = name or func.__name__
+            self._commands.update({func_name: func})
+            return func
+
+        return decorator
+
+    def callback(self, name: Optional[str] = None) -> Callable:
+        """Decorator to register a callback
+
+        Parameters
+        ----------
+        name: Optional[str]
+            The name for the command
+
+        Returns
+        -------
+        Callable
+            The decorated function
+        """
+
+        def decorator(func: Callable) -> Callable:
+            func_name = name or func.__name__
+            self._callback.update({func_name: func})
+            return func
+
+        return decorator
+
+    def print_commands(self) -> None:
+        """Prints the commands and callbacks"""
+        title = f"{self._name} - {self._version}\n"
+        sys.stdout.write(title)
+        sys.stdout.write("{:=<{}}\n".format("", len(title)))
+        sys.stdout.write("Commands:\n")
+        for command in self._commands:
+            sys.stdout.write(f"{command: >20}\n")
+        sys.stdout.write("Callbacks:\n")
+        for callback in self._callback:
+            name = f"--{callback}"
+            sys.stdout.write(f"{name: >20}\n")
+
+    def __call__(self, *args: Any, **kwargs: Any):
+        """Dunder method to run a given command."""
+        if len(sys.argv) == 1:
+            raise MissingCommandError("No command provided")
+
+        command_name = sys.argv[1]
+        raw_args = sys.argv[2:]
+        if command_name.startswith("--"):
+            command = self._retrieve_callback(command_name.lstrip("--"))
+        else:
+            command = self._retrieve_command(command_name)
+
+        signature = inspect.signature(command)
+        command_args = self._parse_arguments(raw_args, signature=signature)
+        command(**command_args)
+
+    def _parse_arguments(
+        self, raw_args: List, signature: inspect.Signature
+    ) -> Dict:
+        arguments = {}
+        for idx, (name, parameter) in enumerate(signature.parameters.items()):
+            try:
+                raw_value = raw_args[idx]
+            except IndexError as exc:
+                raise MissingArgument(f"Missing argument {name}") from exc
+
+            argument = (
+                raw_value.split("=")[-1] if "=" in raw_value else raw_value
+            )
+            arg_value = parameter.annotation(argument)
+            arguments.update({name: arg_value})
+        return arguments
+
+    def _retrieve_command(self, command_name: str) -> Callable:
+        try:
+            command = self._commands[command_name]
+        except KeyError as exc:
+            sys.stderr.write(f"Command {command_name} not found: {exc}")
+        return command
+
+    def _retrieve_callback(self, command_name: str) -> Callable:
+        try:
+            command = self._callback[command_name]
+        except KeyError as exc:
+            sys.stderr.write(f"Callback {command_name} not found: {exc}")
+        return command