rsh-tools 3.0.1__tar.gz

rsh_tools-3.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Grigory Efimov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rsh_tools-3.0.1/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: rsh-tools
+Version: 3.0.1
+Summary: ssh helpers: sr (root shell with privilege escalation), sExec/pExec (run commands on inventory groups), cloud inventory plugins
+Author: Grigory Efimov
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/fb929/rsh
+Project-URL: Issues, https://github.com/fb929/rsh/issues
+Keywords: ssh,sudo,inventory,aws,gce,ovh,devops
+Classifier: Environment :: Console
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: POSIX
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Systems Administration
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: deepmerge>=1.1
+Requires-Dist: pexpect>=4.8
+Requires-Dist: python-gnupg>=0.5
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: fabric>=3.2
+Requires-Dist: braceexpand>=0.1.7
+Provides-Extra: ovh
+Requires-Dist: ovh>=1.1; extra == "ovh"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# rsh
+scripts for auto login and parallel/successively execute command on linux servers
+
+## description
+* sr - server root
+  login over ssh and escalate privileges
+* sExec - successively execute command on servers list or group
+* pExec - parallel execute command on servers list or group
+
+## install
+```
+pip install rsh-tools
+```
+installs the commands `sr`, `sExec`, `pExec`, `rshInventory` and `rshCompgen`.
+with the OVH inventory plugin: `pip install 'rsh-tools[ovh]'`
+
+bash completion for hosts and groups — add to `~/.bashrc`:
+```
+eval "$(rshCompgen completion)"
+```
+
+### development
+```
+git clone https://github.com/fb929/rsh && cd rsh
+make dev     # pip install --editable '.[dev,ovh]'
+make test    # pytest
+```
+the editable install puts the real commands on PATH, so the checkout is what runs.
+
+### release
+bump `__version__` in `rsh/__init__.py` ([semver](https://semver.org)), then tag and push:
+```
+git tag v3.0.0 && git push origin v3.0.0
+```
+the release workflow checks that the tag matches `__version__`, builds sdist + wheel,
+uploads them to PyPI (secret `PYPI_API_TOKEN`) and attaches them to the GitHub release.
+
+## configure
+### (optional) passwords file for "su" mode
+* create encrypt file .pass.gpg
+* add passwords in format:
+  ```
+  <hostname regex>\t<root password>
+  ```
+
+  example
+  ```
+  .*	myRootPasswd
+  ```
+* configure ~/.rsh.yaml
+  ```
+  sr:
+    escalatePrivilegesCommands:
+      - 'su -m'
+      - 'sudo -s -E'
+  ```
+  commands are tried on the host in the listed order; the first one that gives a root shell wins,
+  if none of them succeeds `sr` exits with an error.
+  the old single-value option `escalatePrivilegesCommand` still works, but is used only when `escalatePrivilegesCommands` is empty
+
+### aws inventory plugin
+* install package [aws-cli](https://github.com/aws/aws-cli)
+* configure ~/.aws/credentials
+
+### ovh inventory plugin
+* `pip install 'rsh-tools[ovh]'`
+* configure ~/.rsh.yaml
+  ```
+  ovhInventory:
+    api:
+      endpoint: ovh-ca
+      application_key: xxx
+      application_secret: yyy
+      consumer_key: xxx
+  ```
+  [create api tokens](https://help.ovhcloud.com/csm/en-api-getting-started-ovhcloud-api?id=kb_article_view&sysparm_article=KB0042777)
+
+### gce inventory plugin
+* install add setup [gcloud cli](https://cloud.google.com/sdk/docs/install)
+
+### all configuration options
+see the defaults in [rsh/config.py](rsh/config.py); `~/.rsh.yaml` (and `/etc/rsh/config.yaml`, `./.config.yaml`) are deep-merged over them

rsh_tools-3.0.1/README.md

@@ -0,0 +1,82 @@
+# rsh
+scripts for auto login and parallel/successively execute command on linux servers
+
+## description
+* sr - server root
+  login over ssh and escalate privileges
+* sExec - successively execute command on servers list or group
+* pExec - parallel execute command on servers list or group
+
+## install
+```
+pip install rsh-tools
+```
+installs the commands `sr`, `sExec`, `pExec`, `rshInventory` and `rshCompgen`.
+with the OVH inventory plugin: `pip install 'rsh-tools[ovh]'`
+
+bash completion for hosts and groups — add to `~/.bashrc`:
+```
+eval "$(rshCompgen completion)"
+```
+
+### development
+```
+git clone https://github.com/fb929/rsh && cd rsh
+make dev     # pip install --editable '.[dev,ovh]'
+make test    # pytest
+```
+the editable install puts the real commands on PATH, so the checkout is what runs.
+
+### release
+bump `__version__` in `rsh/__init__.py` ([semver](https://semver.org)), then tag and push:
+```
+git tag v3.0.0 && git push origin v3.0.0
+```
+the release workflow checks that the tag matches `__version__`, builds sdist + wheel,
+uploads them to PyPI (secret `PYPI_API_TOKEN`) and attaches them to the GitHub release.
+
+## configure
+### (optional) passwords file for "su" mode
+* create encrypt file .pass.gpg
+* add passwords in format:
+  ```
+  <hostname regex>\t<root password>
+  ```
+
+  example
+  ```
+  .*	myRootPasswd
+  ```
+* configure ~/.rsh.yaml
+  ```
+  sr:
+    escalatePrivilegesCommands:
+      - 'su -m'
+      - 'sudo -s -E'
+  ```
+  commands are tried on the host in the listed order; the first one that gives a root shell wins,
+  if none of them succeeds `sr` exits with an error.
+  the old single-value option `escalatePrivilegesCommand` still works, but is used only when `escalatePrivilegesCommands` is empty
+
+### aws inventory plugin
+* install package [aws-cli](https://github.com/aws/aws-cli)
+* configure ~/.aws/credentials
+
+### ovh inventory plugin
+* `pip install 'rsh-tools[ovh]'`
+* configure ~/.rsh.yaml
+  ```
+  ovhInventory:
+    api:
+      endpoint: ovh-ca
+      application_key: xxx
+      application_secret: yyy
+      consumer_key: xxx
+  ```
+  [create api tokens](https://help.ovhcloud.com/csm/en-api-getting-started-ovhcloud-api?id=kb_article_view&sysparm_article=KB0042777)
+
+### gce inventory plugin
+* install add setup [gcloud cli](https://cloud.google.com/sdk/docs/install)
+
+### all configuration options
+see the defaults in [rsh/config.py](rsh/config.py); `~/.rsh.yaml` (and `/etc/rsh/config.yaml`, `./.config.yaml`) are deep-merged over them

rsh_tools-3.0.1/pyproject.toml

@@ -0,0 +1,56 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rsh-tools"
+description = "ssh helpers: sr (root shell with privilege escalation), sExec/pExec (run commands on inventory groups), cloud inventory plugins"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{name = "Grigory Efimov"}]
+requires-python = ">=3.9"
+dynamic = ["version"]
+keywords = ["ssh", "sudo", "inventory", "aws", "gce", "ovh", "devops"]
+classifiers = [
+    "Environment :: Console",
+    "Intended Audience :: System Administrators",
+    "Operating System :: POSIX",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Systems Administration",
+]
+dependencies = [
+    "deepmerge>=1.1",
+    "pexpect>=4.8",
+    "python-gnupg>=0.5",
+    "PyYAML>=6.0",
+    "fabric>=3.2",
+    "braceexpand>=0.1.7",
+]
+
+[project.optional-dependencies]
+ovh = ["ovh>=1.1"]
+dev = ["pytest>=7", "build", "twine"]
+
+[project.urls]
+Homepage = "https://github.com/fb929/rsh"
+Issues = "https://github.com/fb929/rsh/issues"
+
+[project.scripts]
+sr = "rsh.sr:main"
+sExec = "rsh.execute:main_sequential"
+pExec = "rsh.execute:main_parallel"
+rshInventory = "rsh.build_inventory:main"
+rshCompgen = "rsh.compgen:main"
+
+[tool.setuptools]
+packages = ["rsh", "rsh.plugins"]
+
+[tool.setuptools.package-data]
+rsh = ["completion.bash"]
+
+[tool.setuptools.dynamic]
+version = {attr = "rsh.__version__"}
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

rsh_tools-3.0.1/rsh/__init__.py

@@ -0,0 +1,3 @@
+"""rsh - ssh helpers: sr (root shell), sExec/pExec (run a command on hosts), inventory plugins."""
+
+__version__ = '3.0.1'

rsh_tools-3.0.1/rsh/build_inventory.py

@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+"""rshInventory - collect hosts from the enabled inventory plugins and write inventory.yaml."""
+
+import os
+import sys
+import json
+import logging
+import re
+
+import yaml
+
+import rsh.config
+from rsh.plugins import PLUGINS
+
+logger = logging.getLogger('rshInventory')
+
+
+def natural_sort_key(name):
+    """Token-aware natural order: 'web2' < 'web10'; the domain part (e.g. '.aws') is ignored."""
+    left = str(name).split('.', 1)[0]
+    tokens = []
+    for rough in re.split(r'[^A-Za-z0-9]+', left):
+        for part in re.findall(r'[A-Za-z]+|\d+', rough):
+            tokens.append((1, int(part)) if part.isdigit() else (0, part.lower()))
+    return tokens
+
+
+def group_name(*parts):
+    name = '_'.join(part.replace('-', '_') for part in parts)
+    return re.sub(r'\s+', '_', name)
+
+
+def collect_records(cfg):
+    """Run every enabled plugin, return the merged list of normalized records."""
+    records = []
+    for key, plugin_class in PLUGINS.items():
+        if not (cfg.get(key) or {}).get('enable'):
+            logger.debug("plugin '%s' disabled", key)
+            continue
+        logger.debug("running plugin '%s'", key)
+        records.extend(plugin_class(cfg).instances_info())
+    for key in cfg:
+        if key.endswith('Inventory') and key not in PLUGINS:
+            logger.warning("config section '%s' has no matching inventory plugin, known: %s", key, sorted(PLUGINS))
+    return records
+
+
+def build_inventory(records, cfg):
+    """Turn plugin records into the {'hosts': ..., 'groups': ...} structure."""
+    hosts = {}
+    groups = {}
+
+    def add_to_group(name, host):
+        groups.setdefault(name, [])
+        if host not in groups[name]:
+            groups[name].append(host)
+
+    for record in records:
+        host = record.get('host')
+        if not host:
+            logger.error("record without 'host': %s", record)
+            sys.exit(1)
+        module = record.get('rshInventoryModule')
+        settings = cfg.get(module) or {}
+        if host not in hosts:
+            hosts[host] = {'sshHost': record['sshHost']}
+            for field in settings.get('hostInfoFields', []):
+                hosts[host][field] = record.get(field, 'unknown')
+        skip_tags = settings.get('skipTagsForMakeGroups', [])
+        for tag in record.get('tags', []):
+            if tag['Key'] in skip_tags:
+                continue
+            add_to_group(group_name('tag', tag['Key'], tag['Value']), host)
+        add_to_group(group_name('dc', record.get('hosting', 'unknown'), record['dc']), host)
+
+    hosts = {name: hosts[name] for name in sorted(hosts, key=natural_sort_key)}
+    groups = {name: sorted(groups[name], key=natural_sort_key) for name in sorted(groups, key=natural_sort_key)}
+    return {'hosts': hosts, 'groups': groups}
+
+
+def write_inventory(inventory, path):
+    path = os.path.expanduser(path)
+    with open(path, 'w') as f:
+        yaml.dump(inventory, f, default_flow_style=False, sort_keys=False)
+    logger.info("inventory written to '%s': %d hosts, %d groups", path, len(inventory['hosts']), len(inventory['groups']))
+
+
+def main():
+    cfg = rsh.config.cfg
+    logger.debug("cfg=%s", json.dumps(cfg, indent=4))
+    records = collect_records(cfg)
+    logger.debug("records=%s", json.dumps(records, indent=4))
+    inventory = build_inventory(records, cfg)
+    logger.debug("inventory=%s", inventory)
+    write_inventory(inventory, cfg['inventoryFilePath'])
+
+
+if __name__ == "__main__":
+    main()

rsh_tools-3.0.1/rsh/compgen.py

@@ -0,0 +1,40 @@
+# -*- coding: utf-8 -*-
+"""rshCompgen - word lists for bash completion, and the completion script itself."""
+
+import argparse
+import logging
+import sys
+from importlib import resources
+
+import rsh.config
+from rsh.inventory import load_inventory
+
+logger = logging.getLogger('rshCompgen')
+
+
+def completion_script():
+    """Bash completion for sr/sExec/pExec; install with: eval "$(rshCompgen completion)"."""
+    return resources.files('rsh').joinpath('completion.bash').read_text()
+
+
+def parse_args(argv=None):
+    parser = argparse.ArgumentParser(description='bash completion helper for sr, sExec and pExec')
+    parser.add_argument(
+        'what',
+        choices=['hosts', 'groups', 'completion'],
+        help="'hosts' or 'groups' print inventory names, 'completion' prints the bash completion script",
+    )
+    return parser.parse_args(argv)
+
+
+def main(argv=None):
+    args = parse_args(argv)
+    if args.what == 'completion':
+        sys.stdout.write(completion_script())
+        return
+    inventory = load_inventory(rsh.config.cfg)
+    print(' '.join(inventory[args.what].keys()))
+
+
+if __name__ == "__main__":
+    main()

rsh_tools-3.0.1/rsh/completion.bash

@@ -0,0 +1,10 @@
+# bash completion for rsh tools; install with:  eval "$(rshCompgen completion)"
+_rsh_completion_hosts(){
+    COMPREPLY=( $(compgen -W "$(rshCompgen hosts)" -- "$2") )
+}
+_rsh_completion_groups(){
+    COMPREPLY=( $(compgen -W "$(rshCompgen groups)" -- "$2") )
+}
+complete -F _rsh_completion_hosts sr
+complete -F _rsh_completion_groups sExec
+complete -F _rsh_completion_groups pExec

rsh_tools-3.0.1/rsh/config.py

@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import os
+from os.path import expanduser
+import sys
+import yaml
+import re
+import logging
+from deepmerge import always_merger
+
+PROGRAM_NAME = 'rsh'
+HOME_DIR = expanduser("~")
+DEFAULT_CONFIG_FILES = [
+    '/etc/' + PROGRAM_NAME + '/config.yaml',
+    HOME_DIR + '/.' + PROGRAM_NAME + '.yaml',
+    './.config.yaml',
+]
+cfg = {
+    #'logFile': HOME_DIR + '/log/' + PROGRAM_NAME + '/' + PROGRAM_NAME + '.log',
+    'logFile': 'stdout',
+    'logLevel': 'info',
+    'gpgFile': HOME_DIR + '/.pass.gpg',
+    'inventoryFilePath': HOME_DIR + '/inventory.yaml',
+    'sensitiveKeys': [
+        'application_key',
+        'application_secret',
+        'consumer_key',
+    ],
+    'awsInventory': {
+        'enable': True,
+        'regions': [],
+        'sshHostField': 'host',             # field for 'sshHost', variants: host, PublicDnsName, PublicIpAddress, PrivateDnsName, PrivateIpAddress
+        'skipTagsForMakeGroups': [          # these tag 'key' will skip when creating groups
+            'Name',
+            'Description',
+            'description',
+        ],
+        'skipNotRunningInstance': True,     # skipping not running instance
+        'hostInfoFields': [
+            'PrivateIpAddress',
+            'PublicIpAddress',
+        ],
+        'nameSuffix': '.aws',               # suffix for instance names
+    },
+    'ovhInventory': {
+        'enable': True,
+        'api': {
+            'endpoint': 'ovh-ca',           # Endpoint of API OVH
+            'application_key': 'xxx',       # Application Key
+            'application_secret': 'yyy',    # Application Secret
+            'consumer_key': 'zzz',          # Consumer Key
+        },
+        'sshHostField': 'host',             # field for 'sshHost', variants: host, customName, publicIp, internalName
+        'skipTagsForMakeGroups': [],        # these tag 'key' will skip when creating groups
+        'hostInfoFields': [
+            'publicIp',
+            'internalName', # internal name
+            'dc',
+        ],
+    },
+    'gceInventory': {
+        'enable': True,
+        'skipNotRunningInstance': True,     # skipping not running instance
+        'sshHostField': 'host',             # field for 'sshHost', variants: host, PrivateIpAddress, PublicIpAddress
+        'hostInfoFields': [
+            'PrivateIpAddress',
+            'PublicIpAddress',
+            'dc',
+        ],
+        'skipTagsForMakeGroups': [          # these tag 'key' will skip when creating groups
+            'name',
+        ],
+        'nameSuffix': '.gc',               # suffix for instance names
+    },
+    'sr': {
+        'ssh': {
+            'command': 'ssh -Y -o ConnectTimeout=5 -o StrictHostKeyChecking=no',
+        },
+        'pexpect': {
+            'timeout': 120,
+        },
+        'useInventorySshHost': True, # use sshHost from inventory for connect to host
+        'escalatePrivilegesCommands': [],   # commands are tried on the host in the listed order until one gives a root shell,
+                                            # e.g. ['su -m', 'sudo -s -E'];
+                                            # must stay empty here: deepmerge appends lists, so a default value would be added to the user's list
+        'escalatePrivilegesCommand': 'sudo -s -E', # deprecated, used only when 'escalatePrivilegesCommands' is empty; variants: 'su -m', 'sudo -s', 'sudo -s -E'
+        'postLoginCommand': None # command will run after login, used to set up the environment
+    },
+}
+
+# load config files {{
+def load_config_files(cfg, paths):
+    """Deep-merge every existing YAML mapping from paths into cfg, in order."""
+    for path in paths:
+        if not os.path.isfile(path):
+            continue
+        try:
+            with open(path, 'r') as f:
+                loaded = yaml.load(f, Loader=yaml.Loader)
+        except Exception as e:
+            logging.warning("skipping config file '%s': %s", path, e)
+            continue
+        if not isinstance(loaded, dict):
+            logging.warning("skipping config file '%s': top level is not a mapping", path)
+            continue
+        cfg = always_merger.merge(cfg, loaded)
+    return cfg
+
+cfg = load_config_files(cfg, DEFAULT_CONFIG_FILES)
+cfg['logDir'] = os.path.dirname(cfg['logFile']) or '.'
+# }}
+
+# logging {{
+def parse_log_level(name):
+    if re.match(r"^(warn|warning)$", str(name), re.IGNORECASE):
+        return logging.WARNING
+    if re.match(r"^debug$", str(name), re.IGNORECASE):
+        return logging.DEBUG
+    return logging.INFO
+
+
+def find_values(data, keys):
+    """Collect values of the given keys anywhere in a nested dict/list structure."""
+    if isinstance(keys, str):
+        keys = [keys]
+    found = []
+    if isinstance(data, dict):
+        for key, value in data.items():
+            if key in keys:
+                found.append(value)
+            else:
+                found.extend(find_values(value, keys))
+    elif isinstance(data, list):
+        for item in data:
+            found.extend(find_values(item, keys))
+    return found
+
+
+class SensitiveFormatter(logging.Formatter):
+    """Masks sensitive config values (api keys etc.) in log output."""
+
+    def __init__(self, fmt, datefmt, sensitive_values):
+        super().__init__(fmt, datefmt=datefmt)
+        values = [str(value) for value in sensitive_values if value]
+        self.pattern = re.compile('|'.join(re.escape(value) for value in values)) if values else None
+
+    def format(self, record):
+        message = super().format(record)
+        if self.pattern is None:
+            return message
+        return self.pattern.sub('[CLASSIFIED]', message)
+
+
+LOG_FORMAT = '%(asctime)s\t%(name)s\t%(funcName)s\t%(levelname)s\t%(message)s'
+LOG_DATE_FORMAT = '%Y-%m-%dT%H:%M:%S'
+
+
+def setup_logging(cfg):
+    level = parse_log_level(cfg['logLevel'])
+    if level > logging.DEBUG:
+        logging.getLogger("urllib3").setLevel(logging.WARNING)
+        logging.getLogger("requests").setLevel(logging.WARNING)
+    if cfg['logFile'] == 'stdout':
+        logging.basicConfig(level=level, format=LOG_FORMAT, datefmt=LOG_DATE_FORMAT)
+    else:
+        os.makedirs(cfg['logDir'], exist_ok=True)
+        logging.basicConfig(filename=cfg['logFile'], level=level, format=LOG_FORMAT, datefmt=LOG_DATE_FORMAT)
+    formatter = SensitiveFormatter(LOG_FORMAT, LOG_DATE_FORMAT, find_values(cfg, cfg['sensitiveKeys']))
+    for handler in logging.root.handlers:
+        handler.setFormatter(formatter)
+
+setup_logging(cfg)
+# }}