olcf-velocity 0.2.dev0__tar.gz → 0.3.dev0__tar.gz

olcf_velocity-0.3.dev0/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.2
+Name: olcf-velocity
+Version: 0.3.dev0
+Summary: A container build manager
+Project-URL: Homepage, https://github.com/olcf/velocity
+Project-URL: Issues, https://github.com/olcf/velocity/issues
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: pyyaml
+Requires-Dist: networkx
+Requires-Dist: colorama
+Requires-Dist: loguru
+Requires-Dist: typing_extensions
+
+![icon.drawio.png](misc/artwork/icon_name.drawio.png)
+
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/tests.yaml?label=tests)](https://github.com/olcf/velocity/actions/workflows/tests.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/docs.yaml?label=docs)](https://github.com/olcf/velocity/actions/workflows/docs.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/linter.yaml?label=linter)](https://github.com/olcf/velocity/actions/workflows/linter.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/build.yaml?label=build&color=https%3A%2F%2Fgithub.com%2Folcf%2Fvelocity%2Factions%2Fworkflows%2Flinter.yaml)](https://github.com/olcf/velocity/actions/workflows/build.yaml)
+[![GitHub Release](https://img.shields.io/github/v/release/olcf/velocity?label=github&color=%23782D84)](https://github.com/olcf/velocity/releases)
+[![PyPI - Version](https://img.shields.io/pypi/v/olcf-velocity?color=%23FFD242)](https://pypi.org/project/olcf-velocity)
+[![Spack](https://img.shields.io/spack/v/py-olcf-velocity?color=%230F3A80)](https://packages.spack.io/package.html?name=py-olcf-velocity)
+
+-----------------------------------------------------------
+
+## Description
+Velocity is a tool to help with the maintenance of container build scripts on 
+multiple systems, backends (e.g podman or apptainer) and distros.
+
+## Documentation
+See <https://olcf.github.io/velocity/>.
+
+## Installation
+``` commandline
+pip install olcf-velocity
+alias velocity="python3 -m velocity"
+velocity
+```

olcf_velocity-0.3.dev0/README.md

@@ -0,0 +1,25 @@
+![icon.drawio.png](misc/artwork/icon_name.drawio.png)
+
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/tests.yaml?label=tests)](https://github.com/olcf/velocity/actions/workflows/tests.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/docs.yaml?label=docs)](https://github.com/olcf/velocity/actions/workflows/docs.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/linter.yaml?label=linter)](https://github.com/olcf/velocity/actions/workflows/linter.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/build.yaml?label=build&color=https%3A%2F%2Fgithub.com%2Folcf%2Fvelocity%2Factions%2Fworkflows%2Flinter.yaml)](https://github.com/olcf/velocity/actions/workflows/build.yaml)
+[![GitHub Release](https://img.shields.io/github/v/release/olcf/velocity?label=github&color=%23782D84)](https://github.com/olcf/velocity/releases)
+[![PyPI - Version](https://img.shields.io/pypi/v/olcf-velocity?color=%23FFD242)](https://pypi.org/project/olcf-velocity)
+[![Spack](https://img.shields.io/spack/v/py-olcf-velocity?color=%230F3A80)](https://packages.spack.io/package.html?name=py-olcf-velocity)
+
+-----------------------------------------------------------
+
+## Description
+Velocity is a tool to help with the maintenance of container build scripts on 
+multiple systems, backends (e.g podman or apptainer) and distros.
+
+## Documentation
+See <https://olcf.github.io/velocity/>.
+
+## Installation
+``` commandline
+pip install olcf-velocity
+alias velocity="python3 -m velocity"
+velocity
+```

{olcf_velocity-0.2.dev0 → olcf_velocity-0.3.dev0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "olcf-velocity"
-version = "0.2.dev"
+version = "0.3.dev"
 description = "A container build manager"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -51,7 +51,7 @@ exclude = [
 ]
 line-length = 120
 indent-width = 4
-target-version = "py310"
+target-version = "py312"
 
 [tool.ruff.lint]
 select = ["E4", "E7", "E9", "F"]

olcf_velocity-0.3.dev0/src/olcf_velocity.egg-info/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.2
+Name: olcf-velocity
+Version: 0.3.dev0
+Summary: A container build manager
+Project-URL: Homepage, https://github.com/olcf/velocity
+Project-URL: Issues, https://github.com/olcf/velocity/issues
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: pyyaml
+Requires-Dist: networkx
+Requires-Dist: colorama
+Requires-Dist: loguru
+Requires-Dist: typing_extensions
+
+![icon.drawio.png](misc/artwork/icon_name.drawio.png)
+
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/tests.yaml?label=tests)](https://github.com/olcf/velocity/actions/workflows/tests.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/docs.yaml?label=docs)](https://github.com/olcf/velocity/actions/workflows/docs.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/linter.yaml?label=linter)](https://github.com/olcf/velocity/actions/workflows/linter.yaml)
+[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/olcf/velocity/build.yaml?label=build&color=https%3A%2F%2Fgithub.com%2Folcf%2Fvelocity%2Factions%2Fworkflows%2Flinter.yaml)](https://github.com/olcf/velocity/actions/workflows/build.yaml)
+[![GitHub Release](https://img.shields.io/github/v/release/olcf/velocity?label=github&color=%23782D84)](https://github.com/olcf/velocity/releases)
+[![PyPI - Version](https://img.shields.io/pypi/v/olcf-velocity?color=%23FFD242)](https://pypi.org/project/olcf-velocity)
+[![Spack](https://img.shields.io/spack/v/py-olcf-velocity?color=%230F3A80)](https://packages.spack.io/package.html?name=py-olcf-velocity)
+
+-----------------------------------------------------------
+
+## Description
+Velocity is a tool to help with the maintenance of container build scripts on 
+multiple systems, backends (e.g podman or apptainer) and distros.
+
+## Documentation
+See <https://olcf.github.io/velocity/>.
+
+## Installation
+``` commandline
+pip install olcf-velocity
+alias velocity="python3 -m velocity"
+velocity
+```

olcf_velocity-0.3.dev0/src/velocity/__init__.py

@@ -0,0 +1,132 @@
+from colorama import Fore, Style
+from loguru import logger; logger.disable("velocity")  # noqa: E702 # disable logging at the module level
+
+from velocity._config import config  # noqa: E402
+from velocity._graph import ImageRepo  # noqa: E402
+from velocity._build import ImageBuilder  # noqa: E402
+from velocity._print import TextBlock, header_print, indent_print  # noqa: E402
+
+
+# config functions
+def get_system() -> str:
+    """Get the system.
+
+    Returns
+    -------
+    str
+        The current system.
+    """
+
+    return config.get("velocity:system")
+
+
+def set_system(system: str) -> None:
+    """Set the system.
+
+    Parameters
+    ----------
+    system : str
+        New value for system.
+    """
+
+    config.set("velocity:system", str(system))
+
+
+def get_backend() -> str:
+    """Get the backend.
+
+    Returns
+    -------
+    str
+        The current backend.
+    """
+
+    return config.get("velocity:backend")
+
+
+def set_backend(backend: str) -> None:
+    """Set the backend.
+
+    Parameters
+    ----------
+    backend : str
+        New value for backend.
+    """
+
+    config.set("velocity:backend", str(backend))
+
+
+def get_distro() -> str:
+    """Get the distro.
+
+    Returns
+    -------
+    str
+        The current distro.
+    """
+
+    return config.get("velocity:distro")
+
+
+def set_distro(distro: str) -> None:
+    """Set the distro.
+
+    Parameters
+    ----------
+    distro : str
+        New value for distro.
+    """
+
+    config.set("velocity:distro", str(distro))
+
+
+def build(
+    targets: str,
+    name: str = None,
+    dry_run: bool = False,
+    leave_tags: bool = False,
+    verbose: bool = False,
+    clean: bool = False,
+) -> None:
+    """Build an image.
+
+    Parameters
+    ----------
+    targets: str
+        A string of build targets e.g. '`gcc@12.4 rocm@5`'.
+    name: str
+        Name of complete image.
+    dry_run: bool
+        Dry run build system.
+    leave_tags: bool
+        Do not clean up intermediate build tags (only applies to dockerish backends).
+    verbose: bool
+        Print helpful debug/runtime information.
+    clean: bool
+        Remove cached files in the build directory.
+    """
+
+    imageRepo = ImageRepo()
+    for p in config.get("velocity:image_path").strip(":").split(":"):
+        imageRepo.import_from_dir(p)
+
+    # get recipe
+    recipe = imageRepo.create_build_recipe(targets.split())[0]
+
+    # print build specs
+    header_print([TextBlock("Build Order:")])
+    for r in recipe:
+        indent_print([TextBlock(f"{r.name}@{r.version}-{r.id}", fore=Fore.MAGENTA, style=Style.BRIGHT)])
+    print()  # newline
+
+    # prep builder
+    builder = ImageBuilder(
+        recipe, build_name=name, dry_run=dry_run, remove_tags=not leave_tags, verbose=verbose, clean_build_dir=clean
+    )
+
+    # build
+    builder.build()
+
+
+# visible attributes
+__all__ = ["get_system", "set_system", "get_backend", "set_backend", "get_distro", "set_distro", "build"]

{olcf_velocity-0.2.dev0 → olcf_velocity-0.3.dev0}/src/velocity/_backends.py

@@ -1,20 +1,23 @@
 """Velocity backends."""
 
-from re import Match as re_Match, sub as re_sub, match as re_match
-from loguru import logger
-from shutil import which as shutil_which
-from pathlib import Path
 from abc import abstractmethod
-from ._exceptions import (
-    RepeatedSection,
+from pathlib import Path
+from re import Match as re_Match, match as re_match, sub as re_sub
+from shutil import which as shutil_which
+from subprocess import run as subprocess_run
+
+from loguru import logger
+
+from velocity._exceptions import (
+    BackendNotAvailable,
+    BackendNotSupported,
     LineOutsideOfSection,
+    RepeatedSection,
     TemplateSyntaxError,
-    BackendNotSupported,
-    BackendNotAvailable,
 )
-from ._config import config
-from ._graph import Image
-from ._tools import OurABCMeta, trace_function
+from velocity._config import config
+from velocity._graph import Image
+from velocity._tools import OurABCMeta, trace_function
 
 
 @trace_function
@@ -44,6 +47,11 @@ class Backend(metaclass=OurABCMeta):
         "@post",
     ]
 
+    existing_builds_cache: dict = dict()
+
+    name: str = "backend"
+    executable: str = "true"
+
     @classmethod
     @trace_function
     def _get_sections(cls, template: list[str]) -> dict[str, list[str]]:
@@ -89,6 +97,7 @@ class Backend(metaclass=OurABCMeta):
     @trace_function
     def _filter_content(cls, image: Image, text: str) -> str:
         """Filter conditionals and white space from a template line."""
+
         # handle conditionals
         res: re_Match[str] = re_match(r".*(\?\?([\S ]*)\|>(.*)\?\?).*", text)
         if res is not None:
@@ -108,6 +117,7 @@ class Backend(metaclass=OurABCMeta):
     @trace_function
     def _load_template(cls, image: Image, variables: dict[str, str]) -> list[str]:
         """Load a template and parse it."""
+
         template: list[str] = list()
         with open(
             Path(image.path).joinpath("templates", "{}.vtmp".format(image.template)),
@@ -120,12 +130,9 @@ class Backend(metaclass=OurABCMeta):
                     template.append(fcon)
         return template
 
-    def __init__(self, name: str, executable: str) -> None:
-        self.name: str = name
-        self.executable: str = executable
-
     def generate_script(self, image: Image, variables: dict[str, str]) -> list[str]:
-        """Generate a build script e.g. .dockerfile/.def"""
+        """Generate a build script."""
+
         logger.debug("Variables: {}".format(variables))
         template: list[str] = self._load_template(image, variables)
         sections: dict[str, list[str]] = self._get_sections(template)
@@ -179,6 +186,7 @@ class Backend(metaclass=OurABCMeta):
 
     def is_available(self) -> bool:
         """Check if the current system has the requested backend."""
+
         if shutil_which(self.executable) is None:
             return False
         return True
@@ -214,13 +222,14 @@ class Backend(metaclass=OurABCMeta):
     @classmethod
     def _literal_section(cls, contents: list[str]) -> list[str]:
         """Handle literal sections."""
+
         ret: list = [""]
         for ln in contents:
             ret.append(ln.lstrip("|"))
         return ret
 
     @abstractmethod
-    def generate_build_cmd(self, src: str, dest: str, args: list[str] = None) -> str:
+    def generate_build_cmd(self, src: str, dest: str, args: list[str] = None) -> list[str]:
         """Generate CLI command to build."""
 
     @abstractmethod
@@ -243,8 +252,8 @@ class Backend(metaclass=OurABCMeta):
 class Apptainer(Backend):
     """Apptainer backend."""
 
-    def __init__(self):
-        super().__init__(name="apptainer", executable="apptainer")
+    name = "apptainer"
+    executable = "apptainer"
 
     def _from(self, contents: list[str]) -> list[str]:
         ret: list[str] = list()
@@ -270,7 +279,7 @@ class Apptainer(Backend):
                     ret.append("From: {}".format(res["main"]))
                 case _:
                     raise TemplateSyntaxError("Unknown bootstrap type '{}' in @from!".format(res["bootstrap"]))
-        else:   # if the bootstrap type was not specified
+        else:  # if the bootstrap type was not specified
             if re_match(r"^.*\.sif$", res["main"]):
                 logger.debug("Template @from source identified as 'localimage'")
                 ret.append("Bootstrap: localimage")
@@ -335,7 +344,7 @@ class Apptainer(Backend):
     def _entry(self, contents: list[str]) -> list[str]:
         return ["", "%runscript", "{}".format(contents[0])]
 
-    def generate_build_cmd(self, src: str, dest: str, args: list = None) -> str:
+    def generate_build_cmd(self, src: str, dest: str, args: list = None) -> list[str]:
         cmd: list[str] = ["{} build".format(self.executable)]
         # arguments
         if args is not None and len(args) > 0:
@@ -344,7 +353,7 @@ class Apptainer(Backend):
         cmd.append("{}".format(dest))
         # script
         cmd.append("{}".format(src))
-        return " ".join(_ for _ in cmd) + ";"
+        return [" ".join(_ for _ in cmd) + ";"]
 
     def format_image_name(self, path: Path, tag: str) -> str:
         return "{}{}".format(Path.joinpath(path, tag), ".sif" if ".sif" not in tag else "")
@@ -353,9 +362,12 @@ class Apptainer(Backend):
         return "echo"
 
     def build_exists(self, name: str) -> bool:
-        if Path(name).is_file():
-            return True
-        return False
+        if name not in self.existing_builds_cache:
+            if Path(name).is_file():
+                self.existing_builds_cache[name] = True
+            else:
+                self.existing_builds_cache[name] = False
+        return self.existing_builds_cache[name]
 
     def generate_final_image_cmd(self, src: str, dest: str) -> str:
         return "cp {} {}".format(src, dest)
@@ -364,8 +376,8 @@ class Apptainer(Backend):
 class Docker(Backend):
     """Docker backend."""
 
-    def __init__(self):
-        super().__init__(name="docker", executable="docker")
+    name = "docker"
+    executable = "docker"
 
     def _from(self, contents: list[str]) -> list[str]:
         return [f"FROM {contents[0]}"]
@@ -416,7 +428,7 @@ class Docker(Backend):
                 ln += "    "
             ln += alt_cmd
             # add '&& \\' to all but the last line
-            if cmd != contents[-1] and cmd[-1] != '\\': # ignore line that end in an escape
+            if cmd != contents[-1] and cmd[-1] != "\\":  # ignore line that end in an escape
                 ln += " && \\"
             ret.append(ln)
         return ret
@@ -433,7 +445,7 @@ class Docker(Backend):
             else:
                 # indent following lines
                 ln += "    "
-            ln += f"{parts[0]}=\"{env.lstrip(parts[0]).strip(' ')}\""
+            ln += f'{parts[0]}="{env.lstrip(parts[0]).strip(" ")}"'
             # add '\\' to all but the last line
             if env != contents[-1]:
                 ln += " \\"
@@ -454,7 +466,7 @@ class Docker(Backend):
             else:
                 # indent following lines
                 ln += "    "
-            ln += f"{parts[0]}=\"{label.lstrip(parts[0]).strip(' ')}\""
+            ln += f'{parts[0]}="{label.lstrip(parts[0]).strip(" ")}"'
             # add '\\' to all but the last line
             if label != contents[-1]:
                 ln += " \\"
@@ -464,7 +476,7 @@ class Docker(Backend):
     def _entry(self, contents: list[str]) -> list[str]:
         return ["", "ENTRYPOINT {}".format(contents[0].split())]
 
-    def generate_build_cmd(self, src: str, dest: str, args: list = None) -> str:
+    def generate_build_cmd(self, src: str, dest: str, args: list = None) -> list[str]:
         cmd: list[str] = ["{} build".format(self.executable)]
         # arguments
         if args is not None and len(args) > 0:
@@ -475,7 +487,7 @@ class Docker(Backend):
         cmd.append("-t {}".format(dest))
         # build dir
         cmd.append(".")
-        return " ".join(_ for _ in cmd) + ";"
+        return [" ".join(_ for _ in cmd) + ";"]
 
     def format_image_name(self, path: Path, tag: str) -> str:
         return "{}{}{}".format("localhost/" if "/" not in tag else "", tag, ":latest" if ":" not in tag else "")
@@ -484,39 +496,92 @@ class Docker(Backend):
         return "{} rmi {}".format(self.executable, name)
 
     def build_exists(self, name: str) -> bool:
-        return False
+        if name not in self.existing_builds_cache:
+            res = subprocess_run(
+                f"{self.executable} image ls -n" + " | awk '{print $1\":\"$2}'" + f" | grep {name}",
+                shell=True,
+                capture_output=True,
+            )
+            if res.returncode == 0:
+                self.existing_builds_cache[name] = True
+            else:
+                self.existing_builds_cache[name] = False
+        return self.existing_builds_cache[name]
 
     def generate_final_image_cmd(self, src: str, dest: str) -> str:
         return "{} tag {} {}".format(self.executable, src, dest)
 
 
+class OpenShift(Docker):
+    """Openshift-CI backend. Inherit from Docker because openshift uses docker as the container runtime."""
+
+    name = "openshift"
+    executable = "oc"
+
+    def generate_build_cmd(self, src: str, dest: str, args: list = None) -> list[str]:
+        arguments = " " + " ".join(_ for _ in args) if args is not None else ""
+        cmd: list[str] = [
+            "cp {} {};".format(src, re_sub(r"(script$)", "Dockerfile", src)),  # copy script to Dockerfile
+            "if ! {} get buildconfigs {}; then".format(self.executable, dest),  # create new build config if none exist
+            "    {} new-build {} --name={} --to={}:latest;".format(
+                self.executable, re_sub(r"(/script$)", "", src), dest, dest
+            ),
+            "fi;",
+            "{} start-build {} --from-dir={} --follow{};".format(  # run build
+                self.executable, dest, re_sub(r"(/script$)", "", src), arguments
+            ),
+            "while ! {} get imagetags {}:latest; do".format(self.executable, dest),  # wait for image to be pushed
+            "    sleep 10;",
+            "done;",
+        ]
+        return cmd
+
+    def format_image_name(self, path: Path, tag: str) -> str:
+        return "v-" + tag
+
+    def clean_up_old_image_tag(self, name: str) -> str:
+        return "{} delete buildconfigs {}; {} delete imagestream {};".format(
+            self.executable, name, self.executable, name
+        )
+
+    def build_exists(self, name: str) -> bool:
+        if name not in self.existing_builds_cache:
+            res = subprocess_run(
+                "{} get imagetags {}:latest;".format(self.executable, name), shell=True, capture_output=True
+            )
+            if res.returncode == 0:
+                self.existing_builds_cache[name] = True
+            else:
+                self.existing_builds_cache[name] = False
+        return self.existing_builds_cache[name]
+
+    def generate_final_image_cmd(self, src: str, dest: str) -> str:
+        return "{} tag {}:latest {}:latest;".format(self.executable, src, dest)
+
+
 class Podman(Docker):
     """Podman backend. Inherit from Docker because for our purposes they are the same."""
 
-    def __init__(self):
-        super().__init__()
-        # override name and executable
-        self.name = "podman"
-        self.executable = "podman"
+    name = "podman"
+    executable = "podman"
 
 
 class Singularity(Apptainer):
     """Singularity backend. Inherit from Apptainer because that is what Singularity really is."""
 
-    def __init__(self):
-        super().__init__()
-        # override name and executable
-        self.name = "singularity"
-        self.executable = "singularity"
+    name = "singularity"
+    executable = "singularity"
 
 
 @trace_function
 def get_backend() -> Backend:
-    backend = config.get("velocity:backend")
+    backend = config.get("velocity:backend").lower()
     if backend == "apptainer":
         b = Apptainer()
     elif backend == "docker":
         b = Docker()
+    elif backend == "openshift":
+        b = OpenShift()
     elif backend == "podman":
         b = Podman()
     elif backend == "singularity":

{olcf_velocity-0.2.dev0 → olcf_velocity-0.3.dev0}/src/velocity/_build.py

@@ -1,20 +1,23 @@
 """Build velocity images."""
 
-import datetime
-import shutil
-import os
+from datetime import datetime, timedelta
+from os import chdir, cpu_count
+from pathlib import Path
+from platform import processor as arch
+from shutil import copy as shutil_copy, copytree, rmtree
+from threading import Thread
 from timeit import default_timer as timer
-from subprocess import Popen, PIPE
 from queue import SimpleQueue
-from threading import Thread
-from pathlib import Path
+from subprocess import PIPE, Popen
+
 from colorama import Fore, Style
-from platform import processor as arch
-from ._config import config
-from ._graph import Image
-from ._print import header_print, indent_print, TextBlock
-from ._backends import get_backend, Backend
-from ._tools import OurMeta, trace_function
+from loguru import logger
+
+from velocity._config import config
+from velocity._graph import Image
+from velocity._print import TextBlock, header_print, indent_print
+from velocity._backends import Backend, get_backend
+from velocity._tools import OurMeta, trace_function
 
 
 @trace_function
@@ -33,6 +36,7 @@ def read_pipe(pipe: PIPE, topic: SimpleQueue, prefix: str, log: SimpleQueue) ->
 def run(cmd: str, log_file: Path = None, verbose: bool = False) -> None:
     """Run a system command logging all output to a file and print if verbose."""
     # open log file (set to False if none is provided)
+    logger.debug("Running command: {}".format(cmd))
     file = open(log_file, "w") if log_file is not None else False
 
     log = SimpleQueue()
@@ -101,13 +105,19 @@ class ImageBuilder(metaclass=OurMeta):
         self.build_dir = Path(config.get("velocity:build_dir"))
         self.build_dir.mkdir(mode=0o777, parents=True, exist_ok=True)
 
-        self.variables: dict[str, str] = dict()
-        for i in self.build_units:
-            self.variables["__{}__version__".format(i.name)] = i.version.__str__()
-            self.variables["__{}__version_major__".format(i.name)] = i.version.major.__str__()
-            self.variables["__{}__version_minor__".format(i.name)] = i.version.minor.__str__()
-            self.variables["__{}__version_patch__".format(i.name)] = i.version.patch.__str__()
-            self.variables["__{}__version_suffix__".format(i.name)] = i.version.suffix.__str__()
+        self.variables: dict[str, str] = {
+            "__backend__": self.backend_engine.name,
+            "__backend_executable__": self.backend_engine.executable,
+            "__threads__": str(int(cpu_count() * 0.75) if int(cpu_count() * 0.75) < 16 else 16),
+            "__arch__": arch(),
+            "__timestamp__": str(datetime.now()),
+        }
+        for u in self.build_units:
+            self.variables["__{}__version__".format(u.name)] = str(u.version)
+            self.variables["__{}__version_major__".format(u.name)] = str(u.version.major)
+            self.variables["__{}__version_minor__".format(u.name)] = str(u.version.minor)
+            self.variables["__{}__version_patch__".format(u.name)] = str(u.version.patch)
+            self.variables["__{}__version_suffix__".format(u.name)] = str(u.version.suffix)
 
     def build(self) -> None:
         """Launch image builds."""
@@ -118,7 +128,7 @@ class ImageBuilder(metaclass=OurMeta):
         if self.clean_build_dir:
             for entry in self.build_dir.iterdir():
                 if entry.is_dir():
-                    shutil.rmtree(entry)
+                    rmtree(entry)
                 else:
                     entry.unlink()
 
@@ -135,7 +145,7 @@ class ImageBuilder(metaclass=OurMeta):
         tag = str(
             self.build_name
             if self.build_name is not None
-            else "{}__{}-{}".format(
+            else "{}_{}-{}".format(
                 "_".join(f"{bu.name}-{bu.version}" for bu in reversed(self.build_units)),
                 config.get("velocity:system"),
                 config.get("velocity:distro"),
@@ -152,7 +162,7 @@ class ImageBuilder(metaclass=OurMeta):
                 run(self.backend_engine.clean_up_old_image_tag(bn))
 
         # go back to the starting dir
-        os.chdir(pwd)
+        chdir(pwd)
 
     def _build_image(self, unit: Image, src_image: str, name: str):
         """Build an individual image."""
@@ -171,7 +181,7 @@ class ImageBuilder(metaclass=OurMeta):
         # create build dir and go to it
         build_sub_dir = Path.joinpath(self.build_dir, "{}-{}-{}".format(unit.name, unit.version, unit.id))
         build_sub_dir.mkdir(mode=0o744, exist_ok=True)
-        os.chdir(build_sub_dir)
+        chdir(build_sub_dir)
 
         # copy additional files
         if len(unit.files) > 0:
@@ -192,9 +202,9 @@ class ImageBuilder(metaclass=OurMeta):
                         ]
                     )
                 if Path.joinpath(unit.path, "files", entry).is_dir():
-                    shutil.copytree(Path.joinpath(unit.path, "files", entry), Path.joinpath(build_sub_dir, entry))
+                    copytree(Path.joinpath(unit.path, "files", entry), Path.joinpath(build_sub_dir, entry))
                 else:
-                    shutil.copy(Path.joinpath(unit.path, "files", entry), Path.joinpath(build_sub_dir, entry))
+                    shutil_copy(Path.joinpath(unit.path, "files", entry), Path.joinpath(build_sub_dir, entry))
 
         # parse template and create script...
         header_print([TextBlock(unit.id, fore=Fore.RED, style=Style.BRIGHT), TextBlock(": GENERATING SCRIPT ...")])
@@ -207,21 +217,19 @@ class ImageBuilder(metaclass=OurMeta):
             )
 
         # get and update script variables
-        script_variables = unit.variables.copy()
-        script_variables.update({"__name__": unit.name})
-        script_variables.update({"__version__": str(unit.version)})
-        script_variables.update({"__version_major__": str(unit.version.major)})
-        script_variables.update({"__version_minor__": str(unit.version.minor)})
-        script_variables.update({"__version_patch__": str(unit.version.patch)})
-        script_variables.update({"__version_suffix__": str(unit.version.suffix)})
-        script_variables.update({"__timestamp__": str(datetime.datetime.now())})
-        script_variables.update(
-            {"__threads__": str(int(os.cpu_count() * 0.75) if int(os.cpu_count() * 0.75) < 16 else 16)}
-        )
-        script_variables.update({"__arch__": arch()})
+        script_variables = {
+            "__image_id__": unit.id,
+            "__name__": unit.name,
+            "__version__": str(unit.version),
+            "__version_major__": str(unit.version.major),
+            "__version_minor__": str(unit.version.minor),
+            "__version_patch__": str(unit.version.patch),
+            "__version_suffix__": str(unit.version.suffix),
+        }
         if src_image is not None:
             script_variables.update({"__base__": src_image})
         script_variables.update(self.variables)
+        script_variables.update(unit.variables)
 
         script = self.backend_engine.generate_script(unit, script_variables)
         # write out script
@@ -231,6 +239,15 @@ class ImageBuilder(metaclass=OurMeta):
                     indent_print([TextBlock(line, fore=Fore.BLUE, style=Style.DIM)])
                 out_file.writelines(line + "\n")
 
+        # write out variables
+        variables_file = build_sub_dir.joinpath("variables")
+        with open(variables_file, "w") as fo:
+            for k, v in script_variables.items():
+                if v == "None":
+                    fo.write("export {}=''".format(k) + "\n")
+                else:
+                    fo.write("export {}={}".format(k, v) + "\n")
+
         # run prolog & build
         build_cmd = self.backend_engine.generate_build_cmd(
             str(Path.joinpath(build_sub_dir, "script")), name, unit.arguments
@@ -245,12 +262,14 @@ class ImageBuilder(metaclass=OurMeta):
                     TextBlock(": RUNNING PROLOG ... && BUILDING ..."),
                 ]
             )
-            build_contents.append(unit.prolog.strip("\n"))
-            build_contents.append(build_cmd)
-
+            build_contents.extend(["### VARIABLES ###", f"source {variables_file.absolute()}"])
+            build_contents.append("### PROLOG ###")
+            build_contents.extend(unit.prolog.strip("\n").split("\n"))
         else:
             header_print([TextBlock(unit.id, fore=Fore.RED, style=Style.BRIGHT), TextBlock(": BUILDING ...")])
-            build_contents.append(build_cmd)
+
+        build_contents.append("### BUILD ###")
+        build_contents.extend(build_cmd)
 
         with open(build_file_path, "w") as build_file:
             for line in build_contents:
@@ -279,7 +298,7 @@ class ImageBuilder(metaclass=OurMeta):
                 TextBlock(" ("),
                 TextBlock("{}@{}".format(unit.name, unit.version), fore=Fore.MAGENTA, style=Style.BRIGHT),
                 TextBlock(") BUILT ["),
-                TextBlock(str(datetime.timedelta(seconds=round(end - start))), fore=Fore.MAGENTA, style=Style.BRIGHT),
+                TextBlock(str(timedelta(seconds=round(end - start))), fore=Fore.MAGENTA, style=Style.BRIGHT),
                 TextBlock("]"),
             ]
         )

{olcf_velocity-0.2.dev0 → olcf_velocity-0.3.dev0}/src/velocity/_config.py

@@ -3,7 +3,8 @@
 from loguru import logger
 from platform import processor as arch
 from pathlib import Path
-from os import getlogin as get_username, getenv
+from os import getenv
+from getpass import getuser
 from yaml import safe_load as yaml_safe_load
 from ._exceptions import InvalidConfigIdentifier
 from ._tools import OurMeta
@@ -99,6 +100,9 @@ if getenv("VELOCITY_IMAGE_PATH") is not None:
 if getenv("VELOCITY_BUILD_DIR") is not None:
     _config.set("velocity:build_dir", getenv("VELOCITY_BUILD_DIR"))
 
+if getenv("VELOCITY_LOGGING_LEVEL") is not None:
+    _config.set("velocity:logging:level", getenv("VELOCITY_LOGGING_LEVEL"))
+
 # set defaults for un-configured items
 if _config.get("velocity:system", warn_on_miss=False) is None:
     _config.set("velocity:system", arch())
@@ -118,7 +122,7 @@ if _config.get("velocity:image_path", warn_on_miss=False) is None:
     _config.set("velocity:image_path", image_dir.__str__())
 
 if _config.get("velocity:build_dir", warn_on_miss=False) is None:
-    _config.set("velocity:build_dir", Path("/tmp").joinpath(get_username(), "velocity").__str__())
+    _config.set("velocity:build_dir", Path("/tmp").joinpath(getuser(), "velocity").__str__())
 
 # export
 config = _config

{olcf_velocity-0.2.dev0 → olcf_velocity-0.3.dev0}/src/velocity/_tools.py

@@ -1,14 +1,13 @@
 """Misc tools."""
 
+from abc import ABCMeta
 from functools import wraps
-from inspect import isfunction, stack, getmodulename
+from inspect import isfunction
 from loguru import logger
-from re import fullmatch as re_fullmatch
-from abc import ABCMeta
+from re import compile as re_compile
 
 
-# this variable holds the current trace indent
-current_trace_indent: list = [0, ]
+builtin_regex = re_compile(r"^__\w+__$")
 
 
 def trace_function(_function):
@@ -27,46 +26,29 @@ def trace_function(_function):
         # update call number
         wrapper.call_count = wrapper.call_count + 1
 
-        # pre-call update indent
-        current_trace_indent[0] = current_trace_indent[0] + 1
-
-        # get caller info
-        current_stack = stack()[1]
-        calling_lineno: int = current_stack.lineno
-        calling_module: str = "velocity.{}".format(getmodulename(current_stack.filename))
-
-        # construct trace message (note: 'this_call_depth' will always be >= 1)
-        depth_str: str = "|{}|".format(" " * (current_trace_indent[0] * 2))
+        # construct trace message
         q_name_str: str = "function '{}:{} {}'".format(
-            _function.__module__,
-            _function.__code__.co_firstlineno,
-            _function.__qualname__
+            _function.__module__, _function.__code__.co_firstlineno, _function.__qualname__
         )
-        call_number_str: str = "call #{} from {}:{} ".format(wrapper.call_count, calling_module, calling_lineno)
-        argument_names: tuple[str] = _function.__code__.co_varnames[:_function.__code__.co_argcount]
+        call_number_str: str = "call #{} ".format(wrapper.call_count)
+        argument_names: tuple[str] = _function.__code__.co_varnames[: _function.__code__.co_argcount]
         argument_str: str = ""
         for a in range(len(args)):
             if argument_names[a] == "self":
                 argument_str += "self, "
             else:
-                argument_str += "{}: {} ='{}', ".format(argument_names[a], type(args[a]).__name__, args[a])
+                argument_str += "{}: {} = '{}', ".format(argument_names[a], type(args[a]).__name__, args[a])
         for a in argument_names:
             if a in kwargs:
-                argument_str += "{}: {} ='{}', ".format(a, type(kwargs[a]).__name__, kwargs[a])
+                argument_str += "{}: {} = '{}', ".format(a, type(kwargs[a]).__name__, kwargs[a])
         argument_str = "with ({})".format(argument_str)
-        argument_str = argument_str.replace("\x1b", "\\x1b")    # neutralize color escape sequences
+        argument_str = argument_str.replace("\x1b", "\\x1b")  # neutralize color escape sequences
 
         # log trace
-        logger.trace("{} {} {} {}".format(depth_str, q_name_str, call_number_str, argument_str))
+        logger.opt(depth=1).trace("{} {} {}".format(q_name_str, call_number_str, argument_str))
 
-        # call _function
-        result = _function(*args, **kwargs)
-
-        # post-call update indent
-        current_trace_indent[0] = current_trace_indent[0] - 1
-
-        # return results of calling _function
-        return result
+        # call inner function
+        return _function(*args, **kwargs)
 
     # set the wrapper call count
     wrapper.call_count = 0
@@ -78,9 +60,10 @@ def trace_function(_function):
 class OurMeta(type):
     """Metaclass to wrap class methods with @trace_function. We do not trace builtin functions
     in the form __\w+__ except for __init__."""
+
     def __new__(cls, *args, **kwargs):
         for name, value in args[2].items():
-            if isfunction(value) and (not re_fullmatch(r"^__\w+__$", name) or name == "__init__"):
+            if isfunction(value) and (not builtin_regex.fullmatch(name) or name == "__init__"):
                 args[2][name] = trace_function(args[2][name])
         return super().__new__(cls, args[0], args[1], args[2])
 
@@ -88,8 +71,9 @@ class OurMeta(type):
 class OurABCMeta(ABCMeta):
     """Metaclass to wrap class methods with @trace_function and inherit from ABCMeta. We do not trace builtin
     functions in the form __\w+__ except for __init__."""
+
     def __new__(cls, *args, **kwargs):
         for name, value in args[2].items():
-            if isfunction(value) and (not re_fullmatch(r"^__\w+__$", name) or name == "__init__"):
+            if isfunction(value) and (not builtin_regex.fullmatch(name) or name == "__init__"):
                 args[2][name] = trace_function(args[2][name])
         return super().__new__(cls, args[0], args[1], args[2])

olcf_velocity-0.2.dev0/PKG-INFO

@@ -1,31 +0,0 @@
-Metadata-Version: 2.1
-Name: olcf-velocity
-Version: 0.2.dev0
-Summary: A container build manager
-Project-URL: Homepage, https://github.com/olcf/velocity
-Project-URL: Issues, https://github.com/olcf/velocity/issues
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE.txt
-Requires-Dist: pyyaml
-Requires-Dist: networkx
-Requires-Dist: colorama
-Requires-Dist: loguru
-Requires-Dist: typing_extensions
-
-![icon.drawio.png](misc/artwork/icon_name.drawio.png)
-
------------------------------------------------------------
-
-## Description
-Velocity is a tool to help with the maintenance of container build scripts on 
-multiple systems, backends (e.g podman or apptainer) and distros.
-
-## Documentation
-See <https://olcf.github.io/velocity/>.
-
-## Installation
-``` commandline
-pip install olcf-velocity
-alias velocity="python3 -m velocity"
-```

olcf_velocity-0.2.dev0/README.md

@@ -1,16 +0,0 @@
-![icon.drawio.png](misc/artwork/icon_name.drawio.png)
-
------------------------------------------------------------
-
-## Description
-Velocity is a tool to help with the maintenance of container build scripts on 
-multiple systems, backends (e.g podman or apptainer) and distros.
-
-## Documentation
-See <https://olcf.github.io/velocity/>.
-
-## Installation
-``` commandline
-pip install olcf-velocity
-alias velocity="python3 -m velocity"
-```

olcf_velocity-0.2.dev0/src/olcf_velocity.egg-info/PKG-INFO

@@ -1,31 +0,0 @@
-Metadata-Version: 2.1
-Name: olcf-velocity
-Version: 0.2.dev0
-Summary: A container build manager
-Project-URL: Homepage, https://github.com/olcf/velocity
-Project-URL: Issues, https://github.com/olcf/velocity/issues
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE.txt
-Requires-Dist: pyyaml
-Requires-Dist: networkx
-Requires-Dist: colorama
-Requires-Dist: loguru
-Requires-Dist: typing_extensions
-
-![icon.drawio.png](misc/artwork/icon_name.drawio.png)
-
------------------------------------------------------------
-
-## Description
-Velocity is a tool to help with the maintenance of container build scripts on 
-multiple systems, backends (e.g podman or apptainer) and distros.
-
-## Documentation
-See <https://olcf.github.io/velocity/>.
-
-## Installation
-``` commandline
-pip install olcf-velocity
-alias velocity="python3 -m velocity"
-```

olcf_velocity-0.2.dev0/src/velocity/__init__.py

@@ -1,5 +0,0 @@
-from loguru import logger; logger.disable("velocity")  # disable logging at the module level
-from ._config import config
-
-
-__all__ = [config]