automagix 4.0.0.dev1__tar.gz → 4.0.0.dev2__tar.gz

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: automagix
-Version: 4.0.0.dev1
+Version: 4.0.0.dev2
 Summary: Automation wrapper for bash and python commands
 Home-page: https://codeberg.org/vanadinit/automagix
 Author: Johannes Paul
@@ -83,10 +83,6 @@ Automagix requires Python ≥ 3.10.
 pip install automagix
 ```
 
-NOTICE: original `automagix` and `automagix_cmd` share the
-same main entrypoint. To avoid overwriting and confusion,
-you should have only installed **ONE** of them!
-
 # CONFIGURATION
 
 You can specify a path to a configuration YAML file via the
@@ -322,9 +318,14 @@ You can refer to these systems in the command pipeline in multiple ways:
    object (Bundlewrap nodes only, no groups)
 
 **vars** _(associative array)_
-: Define some vars. These are accessible in the command pipeline via
- {varname}. Note: Only valid Python variable names are allowed.
- In python actions you can access these variables directly via `VARS.varname`.
+: Define some vars. Note: Only valid Python variable names are allowed.  
+ To specify a variable type use `type|varname` where *type* is
+ one of *int, float, bool, str* and varname the variable name.
+ Automatix will try to convert an assigned value to the specified type.
+ **Note**: For the boolean type the string "False" and upper/lower-case
+ variations are converted to `False` deviating from Python's normal behavior.  
+ Variables are accessible in the command pipeline via
+ {varname} (converted to *str*) and in python actions via `VARS.varname`.
 
 **secrets** _(associative array)_
 : Define teamvault secrets. Value has to be in this format:
@@ -387,8 +388,10 @@ Here you define the commands automagix shall execute.
  To do this prefix the desired variablename and = before the action
  key, e.g. `myvar=python: NODES.system.hostname`. Be careful when
  working with multiline statements. In **python** the first line is
- likely to set the variable. All variables will be converted to
- strings when used to build commands in following steps.
+ likely to set the variable.  
+ Only for **local** and **remote** action: If a variable type was
+ specified in the **vars** section, Automatix will also try to
+ convert the value here.
  
 **CONDITIONS**: You can define the command only to be executed if
  your condition variable evaluates to "True" in Python. To achieve
@@ -398,6 +401,11 @@ Here you define the commands automagix shall execute.
  string which evaluates to "True" in Python, but empty output will
  evaluate to "False". Use `!?` instead of `?` to invert the condition.
 
+**STATIC_FLAG**: If you do not want values to be replaced in your command,
+ you can suffix the command key with '-' to indicate a static command
+ value, e.g. `python-: f'{this} is not replaced'`. This is  especially
+ useful for Python commands which use curly brackets themselves.
+
 **VALUE**: Your command. Variables will be replaced with Python
  format function. Therefore, use curly brackets to refer to variables,
  systems, secrets and constants.
@@ -413,7 +421,7 @@ In most cases its a good idea to define your command in quotes to
 
 #### Escaping in Pipeline
 
-Because automagix uses Python's format() function:  
+Because automagix uses Python's format() function (not for static commands):  
 `{` -> `{{`  
 `}` ->  `}}`  
 

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/README.md

@@ -47,10 +47,6 @@ Automagix requires Python ≥ 3.10.
 pip install automagix
 ```
 
-NOTICE: original `automagix` and `automagix_cmd` share the
-same main entrypoint. To avoid overwriting and confusion,
-you should have only installed **ONE** of them!
-
 # CONFIGURATION
 
 You can specify a path to a configuration YAML file via the
@@ -286,9 +282,14 @@ You can refer to these systems in the command pipeline in multiple ways:
    object (Bundlewrap nodes only, no groups)
 
 **vars** _(associative array)_
-: Define some vars. These are accessible in the command pipeline via
- {varname}. Note: Only valid Python variable names are allowed.
- In python actions you can access these variables directly via `VARS.varname`.
+: Define some vars. Note: Only valid Python variable names are allowed.  
+ To specify a variable type use `type|varname` where *type* is
+ one of *int, float, bool, str* and varname the variable name.
+ Automatix will try to convert an assigned value to the specified type.
+ **Note**: For the boolean type the string "False" and upper/lower-case
+ variations are converted to `False` deviating from Python's normal behavior.  
+ Variables are accessible in the command pipeline via
+ {varname} (converted to *str*) and in python actions via `VARS.varname`.
 
 **secrets** _(associative array)_
 : Define teamvault secrets. Value has to be in this format:
@@ -351,8 +352,10 @@ Here you define the commands automagix shall execute.
  To do this prefix the desired variablename and = before the action
  key, e.g. `myvar=python: NODES.system.hostname`. Be careful when
  working with multiline statements. In **python** the first line is
- likely to set the variable. All variables will be converted to
- strings when used to build commands in following steps.
+ likely to set the variable.  
+ Only for **local** and **remote** action: If a variable type was
+ specified in the **vars** section, Automatix will also try to
+ convert the value here.
  
 **CONDITIONS**: You can define the command only to be executed if
  your condition variable evaluates to "True" in Python. To achieve
@@ -362,6 +365,11 @@ Here you define the commands automagix shall execute.
  string which evaluates to "True" in Python, but empty output will
  evaluate to "False". Use `!?` instead of `?` to invert the condition.
 
+**STATIC_FLAG**: If you do not want values to be replaced in your command,
+ you can suffix the command key with '-' to indicate a static command
+ value, e.g. `python-: f'{this} is not replaced'`. This is  especially
+ useful for Python commands which use curly brackets themselves.
+
 **VALUE**: Your command. Variables will be replaced with Python
  format function. Therefore, use curly brackets to refer to variables,
  systems, secrets and constants.
@@ -377,7 +385,7 @@ In most cases its a good idea to define your command in quotes to
 
 #### Escaping in Pipeline
 
-Because automagix uses Python's format() function:  
+Because automagix uses Python's format() function (not for static commands):  
 `{` -> `{{`  
 `}` ->  `}}`  
 

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/automagix.py

@@ -78,7 +78,7 @@ class Automagix:
         return command_list
 
     def reload_script(self):
-        self.script = get_script(args=self.env.cmd_args)
+        self.env.script = self.script = get_script(args=self.env.cmd_args)
         self._command_lists = {}  # Clear cache
 
     def print_main_data(self):

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/bash_completion.py

@@ -4,7 +4,7 @@ from argparse import Action, Namespace
 
 from argcomplete import warn
 
-from .helpers import read_yaml, search_script
+from .helpers import read_yaml, search_script, NoSuchScriptError, AmbiguousScriptError
 
 
 def _call(*args, **kwargs):
@@ -52,14 +52,13 @@ class ScriptFieldCompleter:
                 return []
 
             s_file = search_script(name=parsed_args.scriptfile, script_dir=self.script_dir, non_interactive=True)
-            if not s_file:
-                warn('Script not found or multiple options. Cannot complete.')
-                return []
-
             script = read_yaml(s_file)
-            completion = [f'{key}=' for key in script.get(action.dest, {}).keys()]
+            completion = [f'{key.split("|", maxsplit=1)[-1]}=' for key in script.get(action.dest, {}).keys()]
 
             return completion
+        except (NoSuchScriptError, AmbiguousScriptError) as exc:
+            warn(f'{exc}. Cannot complete.')
+            return []
         except Exception as exc:
             warn(f'Shell completion failed: {repr(exc)}')
             return []

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/command.py

@@ -6,10 +6,9 @@ from dataclasses import dataclass
 from shlex import quote
 from time import time
 
-from .colors import italic, yellow
-from .environment import PipelineEnvironment, AttributedDict, AttributedDummyDict
+from .colors import italic, yellow, green, red
+from .environment import PipelineEnvironment, AttributedDict, AttributedDummyDict, ConversionError
 from .progress_bar import draw_progress_bar
-from .config import convert_boolean_from_input
 
 PERSISTENT_VARS = PVARS = AttributedDict()
 
@@ -81,6 +80,13 @@ class Command:
         for key, value in cmd.items():
             self.orig_key = key
             self.condition_var, self.assignment_var, self.key = parse_key(key=key)
+
+            if self.key.endswith('-'):
+                self.key = self.key[:-1]
+                self.static = True
+            else:
+                self.static = False
+
             if isinstance(value, dict):
                 # We need this workaround because the yaml lib returns a dictionary instead of a string,
                 # if there is nothing but a variable in the command. Alternative is to use quotes in the script yaml.
@@ -116,6 +122,9 @@ class Command:
         return 'localhost'
 
     def get_resolved_value(self, dummy: bool = False):
+        if self.static:
+            return self.value
+
         variables = self.env.vars.copy()
         variables['CONST'] = ConstantsWrapper(self.env.config['constants'])
         variables['SYSTEMS'] = SystemsWrapper(self.env.systems)
@@ -128,28 +137,41 @@ class Command:
 
     def show_and_change_variables(self):
         print()
-        self.env.LOG.info('Variables:')
+        legend = f'definded type: {italic(green("match"))} - {italic(red("mismatch"))} - {italic(yellow("undefined"))}'
+        self.env.LOG.info(f'--- Variables ({legend}) ---')
+        print()
         keylen = max([len(key) for key in self.env.vars.keys()])
         typelen = len(italic(yellow(''))) + 4
         for key, value in self.env.vars.items():
-            self.env.LOG.info(f" {key:<{keylen+1}}:{italic(yellow(type(value).__name__)):<{typelen}} {value}")
+            type_name = type(value).__name__
+            if key in self.env.script.get('_var_types'):
+                colored_type = green(type_name) if type_name == self.env.script['_var_types'][key] else red(type_name)
+            else:
+                colored_type = yellow(type_name)
+
+            self.env.LOG.info(f" {key:<{keylen + 1}}:{italic(colored_type):<{typelen}} {value}")
         print()
         self.env.LOG.info('To change/set variable write variable + "=" followed by value.')
         self.env.LOG.info('Example: var1=xyz')
         print()
         self.env.LOG.info('You can only change 1 variable at a time. Repeat if necessary.')
-        self.env.LOG.info('Notice: All values are strings here! Use python debugging shell '
-                          'and the VARS dictionary to assign values with other types.')
+        print()
+        self.env.LOG.info('Notice: All values are strings here by default!'
+                          ' If specified in the vars section the value will be converted.'
+                          ' Use Python debug console and the VARS dictionary to assign'
+                          ' values with other types or more complex operations.')
         print()
         self.env.LOG.info('To not change anything just press "ENTER".')
         answer = self.env.interact('\n', progress_portion=self.progress_portion)
         try:
             key, value = answer.split('=', maxsplit=1)
-            self.env.vars[key.strip()] = value.strip()
+            self.env.set_var(key=key.strip(), value=value.strip())
             self.env.LOG.info(f'Variable {key.strip()} = {value.strip()}')
         except ValueError:
             if answer:
                 self.env.LOG.error('Input could not be parsed.')
+        except ConversionError as exc:
+            self.env.LOG.error(exc)
         self.print_command()
         print()
 
@@ -436,9 +458,9 @@ class Command:
             )
             output = proc.stdout.decode(self.env.config["encoding"])
             assigned_value = output.rstrip('\r\n')
-            self.env.vars[self.assignment_var] = convert_boolean_from_input(assigned_value)
+            self.env.set_var(key=self.assignment_var, value=assigned_value)
             hint = ' (trailing newline removed)' if (output.endswith('\n') or output.endswith('\r')) else ''
-            self.env.LOG.info(f'Variable {self.assignment_var} = "{self.env.vars[self.assignment_var]}"{hint}')
+            self.env.LOG.info(f'Variable {self.assignment_var} = "{assigned_value}"{hint}')
         else:
             proc = subprocess.run(
                 cmd,

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/command_test.py

@@ -142,18 +142,21 @@ def test__show_and_change_variables():
         'cond': '{cond}',
         'cond2': '{cond2}',
         'some_var': '{some_var}',
+        'myvar2': 5.,
+        'mybool': False,
     }
 
     with mock.patch.object(
             cmd.env,
             'interact',
-            side_effect=['var1=xyz', 'myvar=hallo', ' cond2  =  !dgkls=432 \n\t  ']
+            side_effect=['var1=xyz', 'myvar=hallo', ' cond2  =  !dgkls=432 \n\t  ', 'mybool=a']
     ) as mock_interact:
         cmd.show_and_change_variables()
         cmd.show_and_change_variables()
         cmd.show_and_change_variables()
+        cmd.show_and_change_variables()
 
-    assert mock_interact.call_count == 3
+    assert mock_interact.call_count == 4
     assert cmd.env.vars == {
         'a': '{a}',
         'myvar': 'hallo',
@@ -161,4 +164,6 @@ def test__show_and_change_variables():
         'cond2': '!dgkls=432',
         'var1': 'xyz',
         'some_var': '{some_var}',
+        'myvar2': 5.,
+        'mybool': True,
     }

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/config.py

@@ -166,11 +166,7 @@ def _overwrite(script: dict, key: str, data: list[str]):
     script.setdefault(key, {})
     for item in data:
         k, v = item.split('=')
-        # Only convert booleans in vars, not in systems or secrets
-        if key == 'vars':
-            script[key][k] = convert_boolean_from_input(v)
-        else:
-            script[key][k] = v
+        script[key][k] = v
 
 
 def _tupelize(string) -> tuple:
@@ -183,30 +179,13 @@ def _tupelize(string) -> tuple:
     return tuple(result)
 
 
-def get_script_path(name: str):
-    s_file = name
-    LOG.debug(f'Scriptfile input: {s_file}')
-    # First try relative path
-    if not os.path.isfile(s_file):
-        LOG.debug('Script not found at relative path')
-        s_file = f'{SCRIPT_DIR}/{name}'
-    # Second try relative path from SCRIPT_DIR
-    if not os.path.isfile(s_file):
-        LOG.debug('Script not found at relative path from SCRIPT_DIR')
-        # Third search and offer selection
-        s_file = search_script(name=name, script_dir=SCRIPT_DIR)
-    if not s_file:
-        LOG.debug('Script not found at all.')
-        raise ValueError('Script not found')
-    LOG.notice(f'Script found at {s_file}.')
-    return s_file
-
-
 def get_script(args: argparse.Namespace) -> dict:
-    s_file = get_script_path(name=args.scriptfile)
+    s_file = search_script(name=args.scriptfile, script_dir=SCRIPT_DIR)
+    LOG.notice(f'Script found at {s_file}.')
 
     try:
         script = read_yaml(s_file)
+        collect_var_types(script)
         validate_script(script)
     except Exception:
         LOG.exception('Script validation failed! Please fix syntax before retrying!')
@@ -227,6 +206,20 @@ def get_script(args: argparse.Namespace) -> dict:
     return script
 
 
+def collect_var_types(script: dict):
+    script['_var_types'] = {}
+    for variable_name in list(script.get('vars')):
+        if match := re.match(r'(\w+)\|(.*)', variable_name):
+            v_name = match.group(2)
+            v_type = match.group(1)
+            if v_type not in ['int', 'float', 'bool', 'str']:
+                raise ValidationError(f'[vars:{v_name}] Unknown type "{v_type}" not allowed.'
+                                      f' Known types are "int", "float", "bool", "str".')
+            script['_var_types'][v_name] = v_type
+            script['vars'][v_name] = script['vars'][variable_name]
+            del script['vars'][variable_name]
+
+
 def check_deprecated_syntax(ckey: str, entry: str, script: dict, prefix: str) -> int:
     warn = 0
     if isinstance(entry, dict):
@@ -389,15 +382,7 @@ def update_script_from_row(row: dict, script: dict, index: int):
             raise ValidationError(
                 f'First row in CSV: Field name is \'{key_type}\','
                 f' but has to be one of {list(SCRIPT_FIELDS.keys())}.')
-
-        script[key_type][key_name] = convert_boolean_from_input(value=value)
-
-
-def convert_boolean_from_input(value: str) -> bool | str:
-    """Convert "true"/"false" strings to boolean, leave everything else as is."""
-    if isinstance(value, str) and value.lower() in ('false', 'true'):
-        return value.lower() == 'true'
-    return value
+        script[key_type][key_name] = value
 
 
 class UnknownSecretTypeException(Exception):

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/environment.py

@@ -1,6 +1,7 @@
 from argparse import Namespace
 from logging import getLogger
 from pathlib import Path
+from typing import Any
 
 from .config import init_logger
 from .helpers import empty_queued_input_data
@@ -46,7 +47,10 @@ class PipelineEnvironment:
     ):
         self.config = config
         self.script = script
-        self.vars = AttributedDict(variables)
+        self.vars = AttributedDict()
+        for k, v in variables.items():
+            self.set_var(key=k, value=v)
+
         self.batch_index = batch_index
         self.cmd_args = cmd_args
 
@@ -83,3 +87,29 @@ class PipelineEnvironment:
         if progress_portion is not None and self.config['progress_bar']:
             draw_progress_bar(progress_portion)
         return answer
+
+    def set_var(self, key: str, value: Any) -> None:
+        if key not in self.script.get('_var_types', {}):
+            self.vars[key] = value
+            return
+
+        var_type = self.script['_var_types'][key]
+        try:
+            match var_type:
+                case 'int':
+                    self.vars[key] = int(value)
+                case 'float':
+                    self.vars[key] = float(value)
+                case 'bool':
+                    if isinstance(value, str) and value.lower() == 'false':
+                        self.vars[key] = False
+                        return
+                    self.vars[key] = bool(value)
+                case 'str':
+                    self.vars[key] = str(value)
+        except ValueError as exc:
+            raise ConversionError(f'Variable conversion failed: Could not convert {key}:"{value}" to {var_type}') from exc
+
+
+class ConversionError(Exception):
+    pass

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix/helpers.py

@@ -1,10 +1,11 @@
-import os
+from os import walk, mkdir, rmdir
+from os.path import abspath, join, isfile
 from sys import stdin
+from termios import tcflush, TCIFLUSH
 from time import sleep
 from typing import List
 
 import yaml
-from termios import tcflush, TCIFLUSH
 
 yaml.warnings({'YAMLLoadWarning': False})
 
@@ -42,15 +43,27 @@ def selector(entries: List[tuple], message: str = 'Found multiple entries, pleas
 
 
 def search_script(name: str, script_dir: str, non_interactive: bool = False) -> str | None:
-    paths = []
-    for dirpath, dirnames, filenames in os.walk(script_dir):
+    paths: set[tuple[str, str]] = set()  # Second str is the label for the selector
+    if isfile(name):
+        return abspath(name)
+    for dirpath, dirnames, filenames in walk(script_dir):
         for filename in filenames:
             if filename == name:
-                path = str(os.path.join(dirpath, filename))
-                paths.append((path, path))  # Second one is the label for the selector
-    if non_interactive and len(paths) != 1:
-        return None
-    return selector(entries=paths, message='Script found at multiple locations. Please choose:')
+                path = str(abspath(join(dirpath, filename)))
+                paths.add((path, path))
+    if not paths:
+        raise NoSuchScriptError('Script not found')
+    if non_interactive and len(paths) > 1:
+        raise AmbiguousScriptError('Multiple scripts found')
+    return selector(entries=list(paths), message='Script found at multiple locations. Please choose:')
+
+
+class NoSuchScriptError(Exception):
+    pass
+
+
+class AmbiguousScriptError(Exception):
+    pass
 
 
 class FileWithLock:
@@ -72,7 +85,7 @@ class FileWithLock:
 def get_lock(file_path: str):
     while True:
         try:
-            os.mkdir(f'{file_path}.lock')
+            mkdir(f'{file_path}.lock')
         except FileExistsError:
             sleep(1)
             continue
@@ -80,4 +93,4 @@ def get_lock(file_path: str):
 
 
 def release_lock(file_path: str):
-    os.rmdir(f'{file_path}.lock')
+    rmdir(f'{file_path}.lock')

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/automagix.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: automagix
-Version: 4.0.0.dev1
+Version: 4.0.0.dev2
 Summary: Automation wrapper for bash and python commands
 Home-page: https://codeberg.org/vanadinit/automagix
 Author: Johannes Paul
@@ -83,10 +83,6 @@ Automagix requires Python ≥ 3.10.
 pip install automagix
 ```
 
-NOTICE: original `automagix` and `automagix_cmd` share the
-same main entrypoint. To avoid overwriting and confusion,
-you should have only installed **ONE** of them!
-
 # CONFIGURATION
 
 You can specify a path to a configuration YAML file via the
@@ -322,9 +318,14 @@ You can refer to these systems in the command pipeline in multiple ways:
    object (Bundlewrap nodes only, no groups)
 
 **vars** _(associative array)_
-: Define some vars. These are accessible in the command pipeline via
- {varname}. Note: Only valid Python variable names are allowed.
- In python actions you can access these variables directly via `VARS.varname`.
+: Define some vars. Note: Only valid Python variable names are allowed.  
+ To specify a variable type use `type|varname` where *type* is
+ one of *int, float, bool, str* and varname the variable name.
+ Automatix will try to convert an assigned value to the specified type.
+ **Note**: For the boolean type the string "False" and upper/lower-case
+ variations are converted to `False` deviating from Python's normal behavior.  
+ Variables are accessible in the command pipeline via
+ {varname} (converted to *str*) and in python actions via `VARS.varname`.
 
 **secrets** _(associative array)_
 : Define teamvault secrets. Value has to be in this format:
@@ -387,8 +388,10 @@ Here you define the commands automagix shall execute.
  To do this prefix the desired variablename and = before the action
  key, e.g. `myvar=python: NODES.system.hostname`. Be careful when
  working with multiline statements. In **python** the first line is
- likely to set the variable. All variables will be converted to
- strings when used to build commands in following steps.
+ likely to set the variable.  
+ Only for **local** and **remote** action: If a variable type was
+ specified in the **vars** section, Automatix will also try to
+ convert the value here.
  
 **CONDITIONS**: You can define the command only to be executed if
  your condition variable evaluates to "True" in Python. To achieve
@@ -398,6 +401,11 @@ Here you define the commands automagix shall execute.
  string which evaluates to "True" in Python, but empty output will
  evaluate to "False". Use `!?` instead of `?` to invert the condition.
 
+**STATIC_FLAG**: If you do not want values to be replaced in your command,
+ you can suffix the command key with '-' to indicate a static command
+ value, e.g. `python-: f'{this} is not replaced'`. This is  especially
+ useful for Python commands which use curly brackets themselves.
+
 **VALUE**: Your command. Variables will be replaced with Python
  format function. Therefore, use curly brackets to refer to variables,
  systems, secrets and constants.
@@ -413,7 +421,7 @@ In most cases its a good idea to define your command in quotes to
 
 #### Escaping in Pipeline
 
-Because automagix uses Python's format() function:  
+Because automagix uses Python's format() function (not for static commands):  
 `{` -> `{{`  
 `}` ->  `}}`  
 

{automagix-4.0.0.dev1 → automagix-4.0.0.dev2}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setup(
     name='automagix',
-    version='4.0.0.dev1',
+    version='4.0.0.dev2',
     description='Automation wrapper for bash and python commands',
     keywords=['bash', 'shell', 'command', 'automation', 'process', 'wrapper', 'devops', 'system administration'],
     long_description=long_description,