cleepcli 1.43.4__tar.gz → 1.43.7__tar.gz

cleepcli-1.43.7/PKG-INFO

@@ -0,0 +1,30 @@
+Metadata-Version: 2.4
+Name: cleepcli
+Version: 1.43.7
+Summary: Cleep-cli helps developers to build great Cleep applications from command line.
+Home-page: https://github.com/CleepDevice/cleep-cli
+Author: Tanguy Bonneau
+Author-email: tanguy.bonneau@gmail.com
+Maintainer: Tanguy Bonneau
+Maintainer-email: tanguy.bonneau@gmail.com
+License-File: LICENSE
+Requires-Dist: Click<9.0,>=8.0
+Requires-Dist: watchdog<4.0.0,>=3.0.0
+Requires-Dist: Sphinx<6.0.0,>=5.0.0
+Requires-Dist: sphinx-rtd-theme>=1.0.0
+Requires-Dist: CProfileV>=1.0.7
+Requires-Dist: PyGithub==1.54.1
+Requires-Dist: pylint<3.0.0,>=2.11.1
+Requires-Dist: semver==3.0.0
+Requires-Dist: coverage==7.2.7
+Requires-Dist: passlib==1.7.4
+Requires-Dist: requests==2.31.0
+Requires-Dist: psutil==5.9.4
+Dynamic: author
+Dynamic: author-email
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: maintainer
+Dynamic: maintainer-email
+Dynamic: requires-dist
+Dynamic: summary

{cleepcli-1.43.4 → cleepcli-1.43.7}/README.md

@@ -1,60 +1,70 @@
 # Cleep-cli
 
 This utility helps developers to build Cleep applications providing some useful commands:
-* Core commands:
-    * `coreget` to clone or pull official Cleep repository.
-    * `coredocs` to generate core documentation in HTML format.
-    * `coresync` to synchronize sources from repository to execution folders.
-    * `coretests` to perform core tests suite.
-    * `coretestscov` to display last core tests code coverage report.
-    * `cpuprof` to run CPU profiler on application execution instance.
-    * `memprof` to run memory profiler on application execution instance.
-    * `reset` clear existing installed Cleep files and install it again.
-* Module commands:
-    * `modcreate` to create module (aka application) skeleton.
-    * `moddelete` to delete installed files of a module.
-    * `moddocs` to generate module documentation.
-    * `modsync` to synchronize sources from module folder to execution folders.
-    * `modtests` to execute module tests.
-    * `modtestscov` to display last module tests code coverage report.
-    * `modbuild` to create application package (.zip)
-    * `modcheckback` to check app backend (python files structure and content)
-    * `modcheckfront` to check app frontend (js files and desc.json)
-    * `modchecktests` to check app tests files
-    * `modcheckscripts` to check app scripts files
-    * `modcheckcode` to check backend code quality (run pylint)
-* Watch commands:
-    * `watch` to watch for repository filesystem changes and automatically update files on execution folders. It also restart backend or/and frontend according to changes.
-* Misc commands:
-    * `version` to get cleep-cli version
+
+- Core commands:
+    - `coreget` to clone or pull official Cleep repository.
+    - `coredocs` to generate core documentation in HTML format.
+    - `coresync` to synchronize sources from repository to execution folders.
+    - `coretests` to perform core tests suite.
+    - `coretestscov` to display last core tests code coverage report.
+    - `cpuprof` to run CPU profiler on application execution instance.
+    - `memprof` to run memory profiler on application execution instance.
+    - `reset` clear existing installed Cleep files and install it again.
+- Module commands:
+    - `modcreate` to create module (aka application) skeleton.
+    - `moddelete` to delete installed files of a module.
+    - `moddocs` to generate module documentation.
+    - `modsync` to synchronize sources from module folder to execution folders.
+    - `modtests` to execute module tests.
+    - `modtestscov` to display last module tests code coverage report.
+    - `modbuild` to create application package (.zip)
+    - `modcheckback` to check app backend (python files structure and content)
+    - `modcheckfront` to check app frontend (js files and desc.json)
+    - `modchecktests` to check app tests files
+    - `modcheckscripts` to check app scripts files
+    - `modcheckcode` to check backend code quality (run pylint)
+- Watch commands:
+    - `watch` to watch for repository filesystem changes and automatically update files on execution folders. It also restart backend or/and frontend according to changes.
+- Misc commands:
+    - `version` to get cleep-cli version
 
 ## Installation
+
 Cleep-cli is automatically installed and managed by [Cleep developer](https://github.com/tangb/cleepmod-developer) application.
 
 If you want to install it manually, execute following command:
+
 > pip install cleepcli
 
 ## Compatibility
+
 This cli is written in Python and is supposed to be used only on Raspbian platform for Cleep application development because it has harcoded paths.
 
 ## Help
+
 Execute this command to get help on this cli:
+
 > cleep-cli --help
 
 ## Watch usage
+
 Launch `watch` cli command to monitor changes on "/root/cleep-dev" directory after getting official Cleep repository using `coreget` command.
 
 Then open your favorite editor on your development computer, configure a remote sync plugin to push local changes on your raspberry pi then cleep-cli will synchronize sources on Cleep execution environment and restart application automatically.
 
 ### Paths
+
 The cli clone repository on `/root/cleep-dev` folder. Modules are in `/root/cleep-dev/modules` folder.
 So if you develop on Cleep core, your editor must sync to `/root/cleep-dev` remote directory while if you're developping only on a module your editor must sync to `/root/cleep-dev/modules/<your module name>` remote directory.
 
 ### Visual studio code
+
 Install Visual studio code [sftp](https://marketplace.visualstudio.com/items?itemName=Natizyskunk.sftp) plugin and configure it to access your raspberry pi:
-* Open VSCode command palette pressing CTRL-SHIFT-P
-* Type "sftp: config"
-* Fill opened file with:
+
+- Open VSCode command palette pressing CTRL-SHIFT-P
+- Type "sftp: config"
+- Fill opened file with:
 
 ```
 {
@@ -81,9 +91,11 @@ Install Visual studio code [sftp](https://marketplace.visualstudio.com/items?ite
 ```
 
 ### Local developments
+
 If you want to develop directly on the raspberry using vim or nano, simply develop on `/root/cleep-dev/` directory. Cleep-cli watcher will synchronize all your modifications.
 
 ## How it works
+
 This cli executes git commands to clone or update repository.
 
 It uses rsync commands to synchronize files.
@@ -92,3 +104,14 @@ It uses pylint to check code quality (it returns a score)
 
 The [watchdog](https://pypi.org/project/watchdog/) python library is used to detect changes on filesystem. According to changed files, it detects if change occurs on frontend or backend files and send commands to restart Cleep application.
 
+## CI (GitHub Actions)
+
+A tag `vX.Y.Z` (or a manual **workflow_dispatch**) builds a **universal** wheel + sdist on `ubuntu-latest` (Python 3.9) and uploads them to PyPI.
+
+GitHub **environment** `actions` (Settings → Environments), secret:
+
+| Secret           | Description                                                 |
+| ---------------- | ----------------------------------------------------------- |
+| `PYPI_API_TOKEN` | PyPI API token (`pypi-...`). Username Twine is `__token__`. |
+
+No Raspberry Pi / Woodpecker needed: the package is pure Python (`py2.py3-none-any`).

cleepcli-1.43.7/cleepcli/ci.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import logging
+import zipfile
+import os
+import glob
+import re
+import requests
+import time
+import urllib.parse
+from . import config
+from .console import Console
+from .check import Check
+from .cleepapi import resolve_rpc_url
+import subprocess
+
+requests.packages.urllib3.disable_warnings()
+
+
+class Ci:
+    """
+    Continuous Integration helpers
+    """
+
+    TESTS_REQUIREMENTS_TXT = "tests/requirements.txt"
+    EXTRACT_DIR = "/tmp/extract"
+    APP_FILENAME_PATTERN = "APP_FILENAME=['\"](.*)['\"]"
+
+    def __init__(self):
+        """
+        Constructor
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.command_url = None
+        self.health_url = None
+
+    def mod_check_package(self, package_path):
+        """
+        Check specified package content
+
+        Args:
+            package_path (string): package path
+
+        Returns:
+            dict: package informations::
+
+            {
+                module_name (str): module name
+                module_version (str): module version
+                has_tests_requirements (bool): True if tests/requirements.txt exists
+            }
+
+        Raises:
+            Exception if something bad detected
+        """
+        self.logger.info('Checking application package "%s"' % package_path)
+        _, module_name, module_version = os.path.basename(package_path).split("_")
+        module_version = module_version.replace(".zip", "")[1:]
+
+        # check raw file
+        if not module_version:
+            raise Exception("Invalid package filename")
+        if not re.match("\d+\.\d+\.\d+", module_version):
+            raise Exception("Invalid package filename")
+        console = Console()
+        resp = console.command('file --keep-going --mime-type "%s"' % package_path)
+        if resp["returncode"] != 0:
+            raise Exception("Unable to check file validity")
+        filetype = resp["stdout"][0].split(": ")[1].strip()
+        self.logger.debug("Filetype=%s" % filetype)
+        if filetype != "application/zip\\012- application/octet-stream":
+            raise Exception("Invalid application package file")
+
+        # check package structure
+        has_tests_requirements = False
+        checks = {
+            "dir_backend": False,
+            "dir_frontend": False,
+            "dir_tests": False,
+            "file_module_json": False,
+            "file_desc_json": False,
+            "file_module_py": False,
+            "file_backend_init_py": False,
+            "file_tests_init_py": False,
+        }
+        with zipfile.ZipFile(package_path, "r") as zp:
+            for zfile in zp.infolist():
+                if zfile.filename.startswith("backend/modules/%s" % module_name):
+                    checks["dir_backend"] = True
+                if zfile.filename.startswith("frontend/js/modules/%s" % module_name):
+                    checks["dir_frontend"] = True
+                if zfile.filename.startswith("tests"):
+                    checks["dir_tests"] = True
+                if zfile.filename == "backend/modules/%s/%s.py" % (
+                    module_name,
+                    module_name,
+                ):
+                    checks["file_module_py"] = True
+                if zfile.filename == "frontend/js/modules/%s/desc.json" % module_name:
+                    checks["file_desc_json"] = True
+                if zfile.filename == "module.json":
+                    checks["file_module_json"] = True
+                if zfile.filename == self.TESTS_REQUIREMENTS_TXT:
+                    has_tests_requirements = True
+                if zfile.filename == "backend/modules/%s/__init__.py" % module_name:
+                    checks["file_backend_init_py"] = True
+                if zfile.filename == "tests/__init__.py":
+                    checks["file_tests_init_py"] = True
+
+            # special case for custom main module filename
+            if checks["file_backend_init_py"] and not checks["file_module_py"]:
+                content = (
+                    zp.read("backend/modules/%s/__init__.py" % module_name)
+                    .decode("utf8")
+                    .strip()
+                )
+                matches = re.findall(self.APP_FILENAME_PATTERN, content)
+                if (
+                    len(matches) == 1
+                    and "backend/modules/%s/%s.py" % (module_name, matches[0])
+                    in zp.namelist()
+                ):
+                    checks["file_module_py"] = True
+
+        self.logger.debug("Checks results: %s" % checks)
+        if not all(checks.values()):
+            raise Exception(
+                "Invalid package structure. Make sure to build it with developer application"
+            )
+
+        return {
+            "module_name": module_name,
+            "module_version": module_version,
+            "has_test_requirements": has_tests_requirements,
+        }
+
+    def mod_install_sources(
+        self, package_path, package_infos, no_compatibility_check=False
+    ):
+        """
+        Install module package (zip archive) sources
+
+        Args:
+            package_path (string): package path
+            package_infos (dict): infos returned by mod_check_package function
+            no_compatitiblity_check (bool): do not check module compatibility (but only deps compat)
+
+        Raises:
+            Exception if error occured
+        """
+        module_name, module_version, has_tests_requirements = package_infos.values()
+        self.logger.info(
+            "Installing application %s v%s..." % (module_name, module_version)
+        )
+
+        cleep_proc = None
+        try:
+            # start cleep (non blocking)
+            self.logger.info("  Starting Cleep...")
+            cleep_proc = subprocess.Popen(
+                ["cleep", "--noro"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
+            )
+            time.sleep(15)
+            self._bind_rpc()
+
+            # make sure to have latest modules.json version
+            self.logger.info("  Refreshing markets...")
+            resp = self._rpc_post(
+                {
+                    "command": "check_modules_updates",
+                    "to": "update",
+                    "timeout": 60.0,
+                }
+            )
+            resp.raise_for_status()
+            resp_json = resp.json()
+            if resp_json["error"]:
+                raise Exception("Check_modules_updates command failed: %s" % resp_json)
+
+            # update "update" module to enjoy bug fixes
+            self.logger.info('  Updating "update" application...')
+            resp = self._rpc_post(
+                {
+                    "command": "update_module",
+                    "to": "update",
+                    "params": {
+                        "module_name": "update",
+                    },
+                    "timeout": 60.0,
+                }
+            )
+            resp.raise_for_status()
+            resp_json = resp.json()
+            self.logger.debug('Update "update" resp: %s' % resp_json)
+            if resp_json["error"]:
+                raise Exception('Updating app "update" failed: %s' % resp_json)
+            if resp_json["data"]:
+                self.__wait_for_cleep_process("update")
+                self.logger.info("  Restarting Cleep...")
+                cleep_proc.kill()
+                cleep_proc = subprocess.Popen(
+                    ["cleep", "--noro"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
+                )
+                time.sleep(15)
+                self._bind_rpc()
+
+            # install module in cleep (it will also install deps)
+            self.logger.info('  Installing "%s" application in Cleep' % module_name)
+            resp = self._rpc_post(
+                {
+                    "command": "install_module",
+                    "to": "update",
+                    "params": {
+                        "module_name": module_name,
+                        "package": package_path,
+                        "no_compatibility_check": no_compatibility_check,
+                    },
+                    "timeout": 60.0,
+                }
+            )
+            resp.raise_for_status()
+            resp_json = resp.json()
+            if resp_json["error"]:
+                raise Exception(
+                    'Installing "%s" app failed: %s' % (module_name, resp_json)
+                )
+            self.__wait_for_cleep_process(module_name)
+
+            # restart cleep
+            self.logger.info("  Restarting Cleep...")
+            cleep_proc.kill()
+            cleep_proc = subprocess.Popen(
+                ["cleep", "--noro"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
+            )
+            time.sleep(15)
+            self._bind_rpc()
+
+            # check module is installed and running
+            self.logger.info("  Checking application is installed...")
+            resp = self._rpc_get(self.health_url)
+            self.logger.info("Health response: %s", resp.json())
+            resp.raise_for_status()
+            self.logger.info("Application and its dependencies successfully installed")
+
+            # install requirements.txt for tests
+            if has_tests_requirements:
+                self.logger.info("Installing tests python dependencies...")
+                console = Console()
+                requirements_txt_path = os.path.join(
+                    config.MODULES_SRC, module_name, self.TESTS_REQUIREMENTS_TXT
+                )
+                self.logger.debug(
+                    'Tests requirements.txt path "%s"', requirements_txt_path
+                )
+                resp = console.command(
+                    'python3 -m pip install --trusted-host pypi.org -r "%s"'
+                    % requirements_txt_path,
+                    900,
+                )
+                self.logger.debug("Resp: %s" % resp)
+                if resp["returncode"] != 0:
+                    self.logger.error(
+                        "Error installing tests requirements.txt: %s", resp
+                    )
+                    raise Exception(
+                        "Error installing tests requirements.txt (killed=%s)"
+                        % resp["killed"]
+                    )
+
+        finally:
+            if cleep_proc:
+                cleep_proc.kill()
+
+    def _bind_rpc(self):
+        """
+        Detect Cleep RPC endpoint (HTTPS first, then HTTP)
+        """
+        base = resolve_rpc_url()
+        self.command_url = urllib.parse.urljoin(base + "/", "command")
+        self.health_url = urllib.parse.urljoin(base + "/", "health")
+
+    def _rpc_post(self, payload):
+        """
+        POST JSON to Cleep /command (self-signed TLS accepted).
+        """
+        return requests.post(self.command_url, json=payload, verify=False)
+
+    def _rpc_get(self, url):
+        """
+        GET Cleep RPC url (self-signed TLS accepted).
+        """
+        return requests.get(url, verify=False)
+
+    def __wait_for_cleep_process(self, module_name):
+        """
+        Wait for end of current Cleep process (install, update...)
+        """
+        while True:
+            time.sleep(1.0)
+            resp = self._rpc_post(
+                {
+                    "command": "get_modules_updates",
+                    "to": "update",
+                    "timeout": 60.0,
+                }
+            )
+            resp.raise_for_status()
+            resp_json = resp.json()
+            if resp_json["error"]:
+                raise Exception("Get_modules_updates command failed")
+            module_updates = resp_json["data"].get(module_name)
+            self.logger.debug("Updates: %s" % module_updates)
+            if not module_updates:
+                raise Exception(
+                    'No "%s" application info in updates data' % module_name
+                )
+            if module_updates["processing"] == False:
+                if module_updates["update"]["failed"]:
+                    raise Exception(
+                        'Application "%s" installation failed' % module_name
+                    )
+                break
+
+    def mod_extract_sources(self, package_path, package_infos):
+        """
+        Extract module package (zip archive) in dev directory
+
+        Args:
+            package_path (string): package path
+            package_infos (dict): infos returned by mod_check_package function
+
+        Raises:
+            Exception if error occured
+        """
+        module_name, module_version, _ = package_infos.values()
+
+        # unzip content
+        self.logger.info(
+            'Extracting archive "%s" to "%s"' % (package_path, self.EXTRACT_DIR)
+        )
+        with zipfile.ZipFile(package_path, "r") as package:
+            package.extractall(self.EXTRACT_DIR)
+
+        # install sources
+        os.makedirs(os.path.join(config.MODULES_SRC, module_name), exist_ok=True)
+        for filepath in glob.glob(self.EXTRACT_DIR + "/**/*.*", recursive=True):
+            if filepath.startswith(os.path.join(self.EXTRACT_DIR, "frontend")):
+                dest = filepath.replace(
+                    os.path.join(
+                        self.EXTRACT_DIR, "frontend/js/modules/%s" % module_name
+                    ),
+                    os.path.join(config.MODULES_SRC, module_name, "frontend"),
+                )
+                self.logger.debug(" -> frontend: %s" % dest)
+            elif filepath.startswith(os.path.join(self.EXTRACT_DIR, "backend")):
+                dest = filepath.replace(
+                    os.path.join(self.EXTRACT_DIR, "backend/modules/%s" % module_name),
+                    os.path.join(config.MODULES_SRC, module_name, "backend"),
+                )
+                self.logger.debug(" -> backend: %s" % dest)
+            elif filepath.startswith(os.path.join(self.EXTRACT_DIR, "tests")):
+                dest = filepath.replace(
+                    os.path.join(self.EXTRACT_DIR, "tests"),
+                    os.path.join(config.MODULES_SRC, module_name, "tests"),
+                )
+                self.logger.debug(" -> tests: %s" % dest)
+            elif filepath.startswith(os.path.join(self.EXTRACT_DIR, "scripts")):
+                dest = filepath.replace(
+                    os.path.join(self.EXTRACT_DIR, "scripts"),
+                    os.path.join(config.MODULES_SRC, module_name, "scripts"),
+                )
+                self.logger.debug(" -> scripts: %s" % dest)
+            else:
+                dest = filepath.replace(
+                    self.EXTRACT_DIR, os.path.join(config.MODULES_SRC, module_name)
+                )
+                self.logger.debug(" -> other: %s" % dest)
+            os.makedirs(os.path.dirname(dest), exist_ok=True)
+            os.rename(filepath, dest)
+
+    def mod_check(self, module_name):
+        """
+        Perform some checkings (see check.py file) for continuous integration
+
+        Args:
+            module_name (string): module name
+
+        Raises:
+            Exception if error occured
+        """
+        check = Check()
+
+        check.check_backend(module_name)
+        check.check_frontend(module_name)
+        check.check_scripts(module_name)
+        check.check_tests(module_name)

{cleepcli-1.43.4 → cleepcli-1.43.7}/cleepcli/cleepapi.py

@@ -10,6 +10,50 @@ import requests
 import json
 import urllib.parse
 
+requests.packages.urllib3.disable_warnings()
+
+# Cleep 0.1+ serves HTTPS by default; older versions listen on HTTP only.
+DEFAULT_RPC_URLS = (
+    'https://127.0.0.1:443',
+    'http://127.0.0.1:80',
+)
+
+
+def resolve_rpc_url(urls=None, timeout=3.0):
+    """
+    Return the first reachable Cleep RPC base URL.
+
+    Tries HTTPS first (Cleep default), then HTTP. Self-signed certificates
+    are accepted. A candidate is considered reachable as soon as a TCP/TLS
+    session is established (any HTTP status).
+
+    Args:
+        urls (tuple): candidate base URLs
+        timeout (float): per-candidate timeout in seconds
+
+    Returns:
+        str: reachable base URL (no trailing slash)
+
+    Raises:
+        Exception: if none of the candidates respond
+    """
+    logger = logging.getLogger('CleepApi')
+    candidates = urls or DEFAULT_RPC_URLS
+    last_error = None
+    for base in candidates:
+        health_url = urllib.parse.urljoin(base.rstrip('/') + '/', 'health')
+        try:
+            logger.debug('Probing Cleep RPC %s', health_url)
+            requests.get(health_url, timeout=timeout, verify=False)
+            logger.info('Using Cleep RPC %s', base)
+            return base.rstrip('/')
+        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout, requests.exceptions.SSLError) as error:
+            logger.debug('Cleep RPC not reachable at %s: %s', base, error)
+            last_error = error
+
+    raise Exception('Unable to reach Cleep RPC (tried %s): %s' % (', '.join(candidates), last_error))
+
+
 class CleepApi():
     """
     Cleep api helper
@@ -19,7 +63,10 @@ class CleepApi():
         self.logger = logging.getLogger(self.__class__.__name__)
 
         if not rpc_url:
-            rpc_url = 'http://0.0.0.0:80'
+            try:
+                rpc_url = resolve_rpc_url()
+            except Exception:
+                rpc_url = DEFAULT_RPC_URLS[0]
         self.logger.debug('RPC url: %s', rpc_url)
 
         self.command_url = urllib.parse.urljoin(rpc_url, "/command")