atex 0.9__tar.gz → 0.11__tar.gz

atex-0.11/.github/workflows/self-tests.yaml

@@ -0,0 +1,54 @@
+name: Sanity self-tests
+
+on:
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  provisioner-testing-farm:
+    runs-on: ubuntu-latest
+    environment:
+      name: SelfTests
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get install -y python3-pytest python3-pytest-xdist
+      - run: pip install .
+      - name: Run tests
+        run: |
+            pytest -s -n 50 tests/provisioner/test_testingfarm.py
+        env:
+          TESTING_FARM_COMPOSE: CentOS-Stream-10
+          TESTING_FARM_API_TOKEN: ${{ secrets.TESTING_FARM_API_TOKEN }}
+  provisioner-podman:
+    runs-on: ubuntu-latest
+    environment:
+      name: SelfTests
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get install -y python3-pytest podman
+      - run: pip install .
+      - name: Run tests
+        run: |
+            pytest -s tests/provisioner/test_podman.py
+  executor:
+    runs-on: ubuntu-latest
+    environment:
+      name: SelfTests
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get install -y python3-pytest
+      - run: pip install .
+      - name: Run tests
+        run: |
+            pytest -s tests/executor
+  fmf:
+    runs-on: ubuntu-latest
+    environment:
+      name: SelfTests
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get install -y python3-pytest
+      - run: pip install .
+      - name: Run tests
+        run: |
+            pytest -s tests/fmf

atex-0.9/README.md → atex-0.11/DEVEL.md

@@ -1,58 +1,34 @@
-# ATEX = Ad-hoc Test EXecutor
+# Misc development notes
 
-A collections of Python APIs to provision operating systems, collect
-and execute [FMF](https://github.com/teemtee/fmf/)-style tests, gather
-and organize their results and generate reports from those results.
+## Contributing
 
-The name comes from a (fairly unique to FMF/TMT ecosystem) approach that
-allows provisioning a pool of systems and scheduling tests on them as one would
-on an ad-hoc pool of thread/process workers - once a worker becomes free,
-it receives a test to run.  
-This is in contrast to splitting a large list of N tests onto M workers
-like N/M, which yields significant time penalties due to tests having
-very varies runtimes.
+TODO - coding style
 
-Above all, this project is meant to be a toolbox, not a silver-plate solution.
-Use its Python APIs to build a CLI tool for your specific use case.  
-The CLI tool provided here is just for demonstration / testing, not for serious
-use - we want to avoid huge modular CLIs for Every Possible Scenario. That's
-the job of the Python API. Any CLI should be simple by nature.
+## Executor and test results
 
----
+TODO: mention that tests output their own JSON + uploaded files
+to some temporary dir, which is then ingested by an Aggregator
+to (potentially) a very different JSON format - the JSON here
+is literally just a format, not a specific kind of data - like
+"INI" doesn't always mean "Midnight Commander config", but a generic
+format useful for many things
 
-THIS PROJECT IS HEAVILY WIP, THINGS WILL MOVE AROUND, CHANGE AND OTHERWISE
-BREAK. DO NOT USE IT (for now).
+TODO: also, test -> results+files --> Aggregator --> more files
+where results+files can have many different keys/values, but
+Aggregators typically only look for a few specific ones (ie. 'note')
 
----
+## Release workflow
 
-## License
+NEVER commit these to git, they are ONLY for the PyPI release.
 
-Unless specified otherwise, any content within this repository is distributed
-under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
-
-## Testing this project
-
-There are some limited sanity tests provided via `pytest`, although:
-
-* Some require additional variables (ie. Testing Farm) and will ERROR
-  without them.
-* Some take a long time (ie. Testing Farm) due to system provisioning
-  taking a long time, so install `pytest-xdist` and run with a large `-n`.
-
-Currently, the recommended approach is to split the execution:
-
-```
-# synchronously, because podman CLI has concurrency issues
-pytest tests/provision/test_podman.py
-
-# in parallel, because provisioning takes a long time
-export TESTING_FARM_API_TOKEN=...
-export TESTING_FARM_COMPOSE=...
-pytest -n 20 tests/provision/test_podman.py
-
-# fast enough for synchronous execution
-pytest tests/fmf
-```
+1. Increase `version = ` in `pyproject.toml`
+1. Tag a new version in the `atex-reserve` repo, push the tag
+1. Point to that tag from `atex/provisioner/testingfarm/api.py`,
+   `DEFAULT_RESERVE_TEST`
+1. ...
+1. `python3 -m build`
+1. `pip install -U twine`
+1. `python3 -m twine upload dist/*`
 
 ## Parallelism and cleanup
 
@@ -146,17 +122,6 @@ Also note that `.reserve()` and `.abort()` could be also called by a context
 manager as `__enter__` and `__exit__`, ie. by a non-threaded caller (running
 everything in the main thread).
 
+## Upcoming API breakages
 
-## Unsorted notes
-
-TODO: codestyle from contest
-
-```
-- this is not tmt, the goal is to make a python toolbox *for* making runcontest
-  style tools easily, not to replace those tools with tmt-style CLI syntax
-
-  - the whole point is to make usecase-targeted easy-to-use tools that don't
-    intimidate users with 1 KB long command line, and runcontest is a nice example
-
-  - TL;DR - use a modular pythonic approach, not a gluetool-style long CLI
-```
+- rename `FMFTests` argument `plan_name` to `plan`

atex-0.11/PKG-INFO

@@ -0,0 +1,86 @@
+Metadata-Version: 2.4
+Name: atex
+Version: 0.11
+Summary: Ad-hoc Test EXecutor
+Project-URL: Homepage, https://github.com/RHSecurityCompliance/atex
+License-Expression: GPL-3.0-or-later
+License-File: COPYING.txt
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: fmf>=1.6
+Requires-Dist: pyyaml
+Requires-Dist: urllib3<3,>=2
+Description-Content-Type: text/markdown
+
+# ATEX = Ad-hoc Test EXecutor
+
+A collections of Python APIs to provision operating systems, collect
+and execute [FMF](https://github.com/teemtee/fmf/)-style tests, gather
+and organize their results and generate reports from those results.
+
+The name comes from a (fairly unique to FMF/TMT ecosystem) approach that
+allows provisioning a pool of systems and scheduling tests on them as one would
+on an ad-hoc pool of thread/process workers - once a worker becomes free,
+it receives a test to run.  
+This is in contrast to splitting a large list of N tests onto M workers
+like N/M, which yields significant time penalties due to tests having
+very varies runtimes.
+
+Above all, this project is meant to be a toolbox, not a silver-plate solution.
+Use its Python APIs to build a CLI tool for your specific use case.  
+The CLI tool provided here is just for demonstration / testing, not for serious
+use - we want to avoid huge modular CLIs for Every Possible Scenario. That's
+the job of the Python API. Any CLI should be simple by nature.
+
+---
+
+## License
+
+Unless specified otherwise, any content within this repository is distributed
+under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
+
+## Environment variables
+
+- `ATEX_DEBUG_TEST`
+  - Set to `1` to print out detailed runner-related trace within the test output
+    stream (as if it was printed out by the test).
+
+## Testing this project
+
+There are some limited sanity tests provided via `pytest`, although:
+
+- Some require additional variables (ie. Testing Farm) and will ERROR
+  without them.
+- Some take a long time (ie. Testing Farm) due to system provisioning
+  taking a long time, so install `pytest-xdist` and run with a large `-n`.
+
+Currently, the recommended approach is to split the execution:
+
+```
+# synchronously, because podman CLI has concurrency issues
+pytest tests/provision/test_podman.py
+
+# in parallel, because provisioning takes a long time
+export TESTING_FARM_API_TOKEN=...
+export TESTING_FARM_COMPOSE=...
+pytest -n 20 tests/provision/test_podman.py
+
+# fast enough for synchronous execution
+pytest tests/fmf
+```
+
+## Unsorted notes
+
+TODO: codestyle from contest
+
+```
+- this is not tmt, the goal is to make a python toolbox *for* making runcontest
+  style tools easily, not to replace those tools with tmt-style CLI syntax
+
+  - the whole point is to make usecase-targeted easy-to-use tools that don't
+    intimidate users with 1 KB long command line, and runcontest is a nice example
+
+  - TL;DR - use a modular pythonic approach, not a gluetool-style long CLI
+```

atex-0.11/README.md

@@ -0,0 +1,70 @@
+# ATEX = Ad-hoc Test EXecutor
+
+A collections of Python APIs to provision operating systems, collect
+and execute [FMF](https://github.com/teemtee/fmf/)-style tests, gather
+and organize their results and generate reports from those results.
+
+The name comes from a (fairly unique to FMF/TMT ecosystem) approach that
+allows provisioning a pool of systems and scheduling tests on them as one would
+on an ad-hoc pool of thread/process workers - once a worker becomes free,
+it receives a test to run.  
+This is in contrast to splitting a large list of N tests onto M workers
+like N/M, which yields significant time penalties due to tests having
+very varies runtimes.
+
+Above all, this project is meant to be a toolbox, not a silver-plate solution.
+Use its Python APIs to build a CLI tool for your specific use case.  
+The CLI tool provided here is just for demonstration / testing, not for serious
+use - we want to avoid huge modular CLIs for Every Possible Scenario. That's
+the job of the Python API. Any CLI should be simple by nature.
+
+---
+
+## License
+
+Unless specified otherwise, any content within this repository is distributed
+under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
+
+## Environment variables
+
+- `ATEX_DEBUG_TEST`
+  - Set to `1` to print out detailed runner-related trace within the test output
+    stream (as if it was printed out by the test).
+
+## Testing this project
+
+There are some limited sanity tests provided via `pytest`, although:
+
+- Some require additional variables (ie. Testing Farm) and will ERROR
+  without them.
+- Some take a long time (ie. Testing Farm) due to system provisioning
+  taking a long time, so install `pytest-xdist` and run with a large `-n`.
+
+Currently, the recommended approach is to split the execution:
+
+```
+# synchronously, because podman CLI has concurrency issues
+pytest tests/provision/test_podman.py
+
+# in parallel, because provisioning takes a long time
+export TESTING_FARM_API_TOKEN=...
+export TESTING_FARM_COMPOSE=...
+pytest -n 20 tests/provision/test_podman.py
+
+# fast enough for synchronous execution
+pytest tests/fmf
+```
+
+## Unsorted notes
+
+TODO: codestyle from contest
+
+```
+- this is not tmt, the goal is to make a python toolbox *for* making runcontest
+  style tools easily, not to replace those tools with tmt-style CLI syntax
+
+  - the whole point is to make usecase-targeted easy-to-use tools that don't
+    intimidate users with 1 KB long command line, and runcontest is a nice example
+
+  - TL;DR - use a modular pythonic approach, not a gluetool-style long CLI
+```

atex-0.11/atex/aggregator/__init__.py

@@ -0,0 +1,62 @@
+import importlib as _importlib
+import pkgutil as _pkgutil
+
+
+class Aggregator:
+    """
+    TODO: generic description, not JSON-specific
+    """
+
+    def ingest(self, platform, test_name, test_results, test_files):
+        """
+        Process 'test_results' (string/Path) for as results reported by a test
+        ran by Executor, along with 'test_files' as files uploaded by that test,
+        aggregating them under 'platform' (string) as 'test_name' (string).
+
+        This is DESTRUCTIVE, the input results/files are consumed in the
+        process.
+        """
+        raise NotImplementedError(f"'ingest' not implemented for {self.__class__.__name__}")
+
+    def start(self):
+        """
+        Start the Aggregator instance, opening any files / allocating resources
+        as necessary.
+        """
+        raise NotImplementedError(f"'start' not implemented for {self.__class__.__name__}")
+
+    def stop(self):
+        """
+        Stop the Aggregator instance, freeing all allocated resources.
+        """
+        raise NotImplementedError(f"'stop' not implemented for {self.__class__.__name__}")
+
+    def __enter__(self):
+        try:
+            self.start()
+            return self
+        except Exception:
+            self.stop()
+            raise
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.stop()
+
+
+_submodules = [
+    info.name for info in _pkgutil.iter_modules(__spec__.submodule_search_locations)
+]
+
+__all__ = [*_submodules, Aggregator.__name__]  # noqa: PLE0604
+
+
+def __dir__():
+    return __all__
+
+
+# lazily import submodules
+def __getattr__(attr):
+    if attr in _submodules:
+        return _importlib.import_module(f".{attr}", __name__)
+    else:
+        raise AttributeError(f"module '{__name__}' has no attribute '{attr}'")

atex-0.11/atex/aggregator/json.py

@@ -0,0 +1,279 @@
+import abc
+import gzip
+import lzma
+import json
+import shutil
+import threading
+from pathlib import Path
+
+from . import Aggregator
+
+
+def _verbatim_move(src, dst):
+    def copy_without_symlinks(src, dst):
+        return shutil.copy2(src, dst, follow_symlinks=False)
+    shutil.move(src, dst, copy_function=copy_without_symlinks)
+
+
+class JSONAggregator(Aggregator):
+    """
+    Collects reported results in a line-JSON output file and uploaded files
+    (logs) from multiple test runs under a shared directory.
+
+    Note that the aggregated JSON file *does not* use the test-based JSON format
+    described by executor/RESULTS.md - both use JSON, but are very different.
+
+    This aggergated format uses a top-level array (on each line) with a fixed
+    field order:
+
+        platform, status, test name, subtest name, files, note
+
+    All these are strings except 'files', which is another (nested) array
+    of strings.
+
+    If 'testout' is present in an input test result, it is prepended to
+    the list of 'files'.
+    If a field is missing in the source result, it is translated to a null
+    value.
+    """
+
+    def __init__(self, target, files):
+        """
+        'target' is a string/Path to a .json file for all ingested
+        results to be aggregated (written) to.
+
+        'files' is a string/Path of the top-level parent for all
+        per-platform / per-test files uploaded by tests.
+        """
+        self.lock = threading.RLock()
+        self.target = Path(target)
+        self.files = Path(files)
+        self.target_fobj = None
+
+    def start(self):
+        if self.target.exists():
+            raise FileExistsError(f"{self.target} already exists")
+        self.target_fobj = open(self.target, "w")
+
+        if self.files.exists():
+            raise FileExistsError(f"{self.files} already exists")
+        self.files.mkdir()
+
+    def stop(self):
+        if self.target_fobj:
+            self.target_fobj.close()
+            self.target_fobj = None
+
+    def _get_test_files_path(self, platform, test_name):
+        """
+        Return a directory path to where uploaded files should be stored
+        for a particular 'platform' and 'test_name'.
+        """
+        platform_files = self.files / platform
+        platform_files.mkdir(exist_ok=True)
+        test_files = platform_files / test_name.lstrip("/")
+        return test_files
+
+    @staticmethod
+    def _modify_file_list(test_files):
+        return test_files
+
+    @staticmethod
+    def _move_test_files(test_files, target_dir):
+        """
+        Move (or otherwise process) 'test_files' as directory of files uploaded
+        by the test, into the pre-computed 'target_dir' location (inside
+        a hierarchy of all files from all tests).
+        """
+        _verbatim_move(test_files, target_dir)
+
+    def _gen_test_results(self, input_fobj, platform, test_name):
+        """
+        Yield complete output JSON objects, one for each input result.
+        """
+        # 'testout' , 'files' and others are standard fields in the
+        # test control interface, see RESULTS.md for the Executor
+        for raw_line in input_fobj:
+            result_line = json.loads(raw_line)
+
+            file_names = []
+            # process the file specified by the 'testout' key
+            if "testout" in result_line:
+                file_names.append(result_line["testout"])
+            # process any additional files in the 'files' key
+            if "files" in result_line:
+                file_names += (f["name"] for f in result_line["files"])
+
+            file_names = self._modify_file_list(file_names)
+
+            output_line = (
+                platform,
+                result_line["status"],
+                test_name,
+                result_line.get("name"),  # subtest
+                file_names,
+                result_line.get("note"),
+            )
+            yield json.dumps(output_line, indent=None)
+
+    def ingest(self, platform, test_name, test_results, test_files):
+        target_test_files = self._get_test_files_path(platform, test_name)
+        if target_test_files.exists():
+            raise FileExistsError(f"{target_test_files} already exists for {test_name}")
+
+        # parse the results separately, before writing any aggregated output,
+        # to ensure that either ALL results from the test are ingested, or none
+        # at all (ie. if one of the result lines contains JSON errors)
+        with open(test_results) as test_results_fobj:
+            output_results = self._gen_test_results(test_results_fobj, platform, test_name)
+            output_json = "\n".join(output_results) + "\n"
+
+        with self.lock:
+            self.target_fobj.write(output_json)
+            self.target_fobj.flush()
+
+        # clean up the source test_results (Aggregator should 'mv', not 'cp')
+        Path(test_results).unlink()
+
+        # if the test_files dir is not empty
+        if any(test_files.iterdir()):
+            self._move_test_files(test_files, target_test_files)
+
+
+class CompressedJSONAggregator(JSONAggregator, abc.ABC):
+    compress_files = False
+    suffix = ""
+    exclude = ()
+
+    @abc.abstractmethod
+    def compressed_open(self, *args, **kwargs):
+        pass
+
+    def start(self):
+        if self.target.exists():
+            raise FileExistsError(f"{self.target_file} already exists")
+        self.target_fobj = self.compressed_open(self.target, "wt", newline="\n")
+
+        if self.files.exists():
+            raise FileExistsError(f"{self.storage_dir} already exists")
+        self.files.mkdir()
+
+    def _modify_file_list(self, test_files):
+        if self.compress_files and self.suffix:
+            return [
+                (name if name in self.exclude else f"{name}{self.suffix}")
+                for name in test_files
+            ]
+        else:
+            return super()._modify_file_list(test_files)
+
+    def _move_test_files(self, test_files, target_dir):
+        if not self.compress_files:
+            super()._move_test_files(test_files, target_dir)
+            return
+
+        for root, _, files in test_files.walk(top_down=False):
+            for file_name in files:
+                src_path = root / file_name
+                dst_path = target_dir / src_path.relative_to(test_files)
+
+                dst_path.parent.mkdir(parents=True, exist_ok=True)
+
+                # skip dirs, symlinks, device files, etc.
+                if not src_path.is_file(follow_symlinks=False) or file_name in self.exclude:
+                    _verbatim_move(src_path, dst_path)
+                    continue
+
+                if self.suffix:
+                    dst_path = dst_path.with_name(f"{dst_path.name}{self.suffix}")
+
+                with open(src_path, "rb") as plain_fobj:
+                    with self.compressed_open(dst_path, "wb") as compress_fobj:
+                        shutil.copyfileobj(plain_fobj, compress_fobj, 1048576)
+
+                src_path.unlink()
+
+            # we're walking bottom-up, so the local root should be empty now
+            root.rmdir()
+
+
+class GzipJSONAggregator(CompressedJSONAggregator):
+    """
+    Identical to JSONAggregator, but transparently Gzips either or both of
+    the output line-JSON file with results and the uploaded files.
+    """
+    def compressed_open(self, *args, **kwargs):
+        return gzip.open(*args, compresslevel=self.level, **kwargs)
+
+    def __init__(
+        self, target, files, *, compress_level=9,
+        compress_files=True, compress_files_suffix=".gz", compress_files_exclude=None,
+    ):
+        """
+        'target' is a string/Path to a .json.gz file for all ingested
+        results to be aggregated (written) to.
+
+        'files' is a string/Path of the top-level parent for all
+        per-platform / per-test files uploaded by tests.
+
+        'compress_level' specifies how much effort should be spent compressing,
+        (1 = fast, 9 = slow).
+
+        If 'compress_files' is True, compress also any files uploaded by tests.
+
+        The 'compress_files_suffix' is appended to any processed test-uploaded
+        files, and the respective 'files' results array is modified with the
+        new file names (as if the test uploaded compressed files already).
+        Set to "" (empty string) to use original file names and just compress
+        them transparently in-place.
+
+        'compress_files_exclude' is a tuple/list of strings (input 'files'
+        names) to skip when compressing. Their names also won't be modified.
+        """
+        super().__init__(target, files)
+        self.level = compress_level
+        self.compress_files = compress_files
+        self.suffix = compress_files_suffix
+        self.exclude = compress_files_exclude or ()
+
+
+class LZMAJSONAggregator(CompressedJSONAggregator):
+    """
+    Identical to JSONAggregator, but transparently compresses (via LZMA/XZ)
+    either or both of the output line-JSON file with results and the uploaded
+    files.
+    """
+    def compressed_open(self, *args, **kwargs):
+        return lzma.open(*args, preset=self.preset, **kwargs)
+
+    def __init__(
+        self, target, files, *, compress_preset=9,
+        compress_files=True, compress_files_suffix=".xz", compress_files_exclude=None,
+    ):
+        """
+        'target' is a string/Path to a .json.xz file for all ingested
+        results to be aggregated (written) to.
+
+        'files' is a string/Path of the top-level parent for all
+        per-platform / per-test files uploaded by tests.
+
+        'compress_preset' specifies how much effort should be spent compressing,
+        (1 = fast, 9 = slow). Optionally ORed with lzma.PRESET_EXTREME to spend
+        even more CPU time compressing.
+
+        If 'compress_files' is True, compress also any files uploaded by tests.
+
+        The 'compress_files_suffix' is appended to any processed test-uploaded
+        files, and the respective 'files' results array is modified with the
+        new file names (as if the test uploaded compressed files already).
+        Set to "" (empty string) to use original file names and just compress
+        them transparently in-place.
+
+        'compress_files_exclude' is a tuple/list of strings (input 'files'
+        names) to skip when compressing. Their names also won't be modified.
+        """
+        super().__init__(target, files)
+        self.preset = compress_preset
+        self.compress_files = compress_files
+        self.suffix = compress_files_suffix
+        self.exclude = compress_files_exclude or ()

{atex-0.9 → atex-0.11}/atex/cli/__init__.py

@@ -27,12 +27,21 @@ import pkgutil
 import argparse
 import logging
 
+from .. import util
+
 
 def setup_logging(level):
+    if level <= util.EXTRADEBUG:
+        fmt = "%(asctime)s %(name)s: %(filename)s:%(lineno)s: %(funcName)s(): %(message)s"
+        # also print urllib3 headers
+        import http.client  # noqa: PLC0415
+        http.client.HTTPConnection.debuglevel = 5
+    else:
+        fmt = "%(asctime)s %(name)s: %(message)s"
     logging.basicConfig(
         level=level,
         stream=sys.stderr,
-        format="%(asctime)s %(name)s: %(message)s",
+        format=fmt,
         datefmt="%Y-%m-%d %H:%M:%S",
     )
 
@@ -53,6 +62,10 @@ def main():
         "--debug", "-d", action="store_const", dest="loglevel", const=logging.DEBUG,
         help="enable extra debugging (logging.DEBUG)",
     )
+    log_grp.add_argument(
+        "--extra-debug", "-D", action="store_const", dest="loglevel", const=util.EXTRADEBUG,
+        help="enable extra debugging (atex.util.EXTRADEBUG)",
+    )
     log_grp.add_argument(
         "--quiet", "-q", action="store_const", dest="loglevel", const=logging.WARNING,
         help="be quiet during normal operation (logging.WARNING)",