atex 0.2__tar.gz → 0.4__tar.gz

atex-0.4/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.4
+Name: atex
+Version: 0.4
+Summary: Ad-hoc Test EXecutor
+Project-URL: Homepage, https://github.com/RHSecurityCompliance/atex
+License-Expression: GPL-3.0-or-later
+License-File: COPYING.txt
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Requires-Dist: fmf>=1.6
+Requires-Dist: urllib3<3,>=2
+Description-Content-Type: text/markdown
+
+# ATEX = Ad-hoc Test EXecutor
+
+A collections of Python APIs to provision operating systems, collect
+and execute [FMF](https://github.com/teemtee/fmf/)-style tests, gather
+and organize their results and generate reports from those results.
+
+The name comes from a (fairly unique to FMF/TMT ecosystem) approach that
+allows provisioning a pool of systems and scheduling tests on them as one would
+on an ad-hoc pool of thread/process workers - once a worker becomes free,
+it receives a test to run.  
+This is in contrast to splitting a large list of N tests onto M workers
+like N/M, which yields significant time penalties due to tests having
+very varies runtimes.
+
+Above all, this project is meant to be a toolbox, not a silver-plate solution.
+Use its Python APIs to build a CLI tool for your specific use case.  
+The CLI tool provided here is just for demonstration / testing, not for serious
+use - we want to avoid huge modular CLIs for Every Possible Scenario. That's
+the job of the Python API. Any CLI should be simple by nature.
+
+---
+
+THIS PROJECT IS HEAVILY WIP, THINGS WILL MOVE AROUND, CHANGE AND OTHERWISE
+BREAK. DO NOT USE IT (for now).
+
+---
+
+## License
+
+Unless specified otherwise, any content within this repository is distributed
+under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
+
+## Unsorted notes
+
+```
+- this is not tmt, the goal is to make a python toolbox *for* making runcontest
+  style tools easily, not to replace those tools with tmt-style CLI syntax
+
+  - the whole point is to make usecase-targeted easy-to-use tools that don't
+    intimidate users with 1 KB long command line, and runcontest is a nice example
+
+  - TL;DR - use a modular pythonic approach, not a modular CLI like tmt
+
+
+- Orchestrator with
+  - add_provisioner(<class>, max_workers=1)   # will instantiate <class> at most max_workers at a time
+  - algo
+    - for all provisioner classes, spawns classes*max_workers as new Threads
+    - waits for any .reserve() to return
+    - creates a new Thread for minitmt, gives it p.get_ssh() details
+      - minitmt will
+        - establish a SSHConn
+        - install test deps, copy test repo over, prepare socket dir on SUT, etc.
+        - run the test in the background as
+            f=os.open('some/test/log', os.WRONLY); subprocess.Popen(..., stdout=f, stderr=f, stdin=subprocess.DEVNULL)
+        - read/process Unix sock results in the foreground, non-blocking,
+          probably calling some Orchestrator-provided function to store results persistently
+        - regularly check Popen proc status, re-accept UNIX sock connection, etc., etc.
+      - minitmt also has some Thread-independent way to .cancel(), killing the proc, closing SSHConn, etc.
+
+  - while waiting for minitmt Threads to finish, to re-assign existing Provisioner instances
+    to new minitmt Threads, .. Orchestrator uses some logic to select, which TestRun
+    would be ideal to run next
+    - TestRun probably has some "fitness" function that returns some priority number
+      when given a Provisioner instance (?) ...
+    - something from minitmt would also have access to the Provisioner instance
+    - the idea is to allow some logic to set "hey I set up nested VM snapshot on this thing"
+      on the Provisioner instance, and if another /hardening/oscap TestRun finds
+      a Provisioner instance like that, it would return high priority
+    - ...
+    - similar to "fitness" like function, we need some "applicability" function
+      - if TestRun is mixed to RHEL-9 && x86_64, we need it to return True
+        for a Provisioner instance that provides RHEL-9 and x86_64, but False otherwise
+
+- basically Orchestrator has
+  - .add_provisioner()
+  - .run_test()  # called with an exclusively-borrowed Provisioner instance
+    - if Provisioner is_alive()==False after .run_test(), instantiate a new one from the same inst.__class__
+    - if test failed and reruns > 0, try run_test() again (or maybe re-queue the test)
+  - .output_result()  # called by run_test() to persistently log a test result
+  - .applicable()  # return True if a passed TestRun is meant for a passed Platform (Provisioner?)
+    - if no TestRun returns True, the Provisioner is .release()d because we don't need it anymore
+  - .fitness()    # return -inf / 0 / +inf with how much should a passed TestRun run on a Provisioner
+  - MAYBE combine applicable() and fitness() into one function, next_test() ?
+    - given the free Provisioner and a list of TestRuns, select which should run next on the Provisioner
+      - if none is chosen, .release() the Provisioner without replacement, continue
+```

atex-0.4/atex/cli/minitmt.py

@@ -0,0 +1,82 @@
+import re
+import pprint
+
+#from .. import util
+from ..minitmt import fmf
+
+
+def _get_context(args):
+    context = {}
+    if args.context:
+        for c in args.context:
+            key, value = c.split('=', 1)
+            context[key] = value
+    return context or None
+
+
+def discover(args):
+    result = fmf.FMFData(args.root, args.plan, context=_get_context(args))
+    for test in result.tests:
+        print(test.name)
+
+
+def show(args):
+    result = fmf.FMFData(args.root, args.plan, context=_get_context(args))
+    for test in result.tests:
+        if re.match(args.test, test.name):
+            pprint.pprint(test.data)
+            break
+    else:
+        print(f"Not reachable via {args.plan} discovery: {args.test}")
+        raise SystemExit(1)
+
+
+def parse_args(parser):
+    parser.add_argument('--root', default='.', help="path to directory with fmf tests")
+    parser.add_argument('--context', '-c', help="tmt style key=value context", action='append')
+    cmds = parser.add_subparsers(
+        dest='_cmd', help="minitmt feature", metavar='<cmd>', required=True,
+    )
+
+    cmd = cmds.add_parser(
+        'discover', aliases=('di',),
+        help="list tests, post-processed by tmt plans",
+    )
+    cmd.add_argument('plan', help="tmt plan to use for discovery")
+
+    cmd = cmds.add_parser(
+        'show',
+        help="show fmf data of a test",
+    )
+    cmd.add_argument('plan', help="tmt plan to use for discovery")
+    cmd.add_argument('test', help="fmf style test regex")
+
+    cmd = cmds.add_parser(
+        'execute', aliases=('ex',),
+        help="run a plan (or test) on a remote system",
+    )
+    grp = cmd.add_mutually_exclusive_group()
+    grp.add_argument('--test', '-t', help="fmf style test regex")
+    grp.add_argument('--plan', '-p', help="tmt plan name (path) inside metadata root")
+    cmd.add_argument('--ssh-identity', '-i', help="path to a ssh keyfile for login")
+    cmd.add_argument('user_host', help="ssh style user@host of the remote")
+
+
+def main(args):
+    if args._cmd in ('discover', 'di'):
+        discover(args)
+    elif args._cmd == 'show':
+        show(args)
+    elif args._cmd in ('execute', 'ex'):
+        #execute(args)
+        print("not implemented yet")
+    else:
+        raise RuntimeError(f"unknown args: {args}")
+
+
+CLI_SPEC = {
+    'aliases': ('tmt',),
+    'help': "simple test executor using atex.minitmt",
+    'args': parse_args,
+    'main': main,
+}

{atex-0.2 → atex-0.4}/atex/cli/testingfarm.py

@@ -1,5 +1,5 @@
 import sys
-#from datetime import datetime
+import pprint
 
 from .. import util
 from .. import testingfarm as tf
@@ -14,6 +14,16 @@ def _get_api(args):
     return tf.TestingFarmAPI(**api_args)
 
 
+def about(args):
+    api = _get_api(args)
+    pprint.pprint(api.about())
+
+
+def whoami(args):
+    api = _get_api(args)
+    pprint.pprint(api.whoami())
+
+
 def composes(args):
     api = _get_api(args)
     comps = api.composes(ranch=args.ranch)
@@ -29,6 +39,11 @@ def get_request(args):
     print(str(request))
 
 
+def cancel(args):
+    api = _get_api(args)
+    api.cancel_request(args.request_id)
+
+
 def search_requests(args):
     api = _get_api(args)
     reply = api.search_requests(
@@ -43,9 +58,6 @@ def search_requests(args):
 
     for req in sorted(reply, key=lambda x: x['created']):
         req_id = req['id']
-        #created_utc = req['created'].partition('.')[0]
-        #created_dt = datetime.fromisoformat(f'{created_utc}+00:00')
-        #created = created_dt.astimezone().isoformat().partition('.')[0]
         created = req['created'].partition('.')[0]
 
         envs = []
@@ -58,29 +70,51 @@ def search_requests(args):
         envs_str = ', '.join(envs)
 
         print(f'{created} {req_id} : {envs_str}')
-        #request = tf.Request(initial_data=req)
-        #print(str(request))
-    #request.update()
-    #print(str(request))
 
 
 def reserve(args):
     util.info(f"Reserving {args.compose} on {args.arch} for {args.timeout} minutes")
 
+    if args.hvm:
+        hardware = {'virtualization': {'is-supported': True}}
+    else:
+        hardware = None
+
     api = _get_api(args)
     res = tf.Reserve(
         compose=args.compose,
         arch=args.arch,
         timeout=args.timeout,
+        hardware=hardware,
         api=api,
     )
     with res as m:
         util.info(f"Got machine: {m}")
-        util.subprocess_run([
-            'ssh', '-q', '-i', m.ssh_key,
-            '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null',
-            f'{m.user}@{m.host}',
-        ])
+        while True:
+            try:
+                res.request.update()
+                res.request.assert_alive()
+            except tf.GoneAwayError as e:
+                print(e)
+                raise SystemExit(1)
+
+            proc = util.subprocess_run([
+                'ssh', '-q', '-i', m.ssh_key,
+                '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null',
+                f'{m.user}@{m.host}',
+            ])
+            if proc.returncode != 0:
+                print(
+                    f"\nssh -i {str(m.ssh_key)} {m.user}@{m.host}\n"
+                    f"terminated with exit code {proc.returncode}\n",
+                )
+                try:
+                    input("Press RETURN to try to reconnect, Ctrl-C to quit ...")
+                except KeyboardInterrupt:
+                    print()
+                    raise
+            else:
+                break
 
 
 def watch_pipeline(args):
@@ -104,12 +138,21 @@ def watch_pipeline(args):
 
 
 def parse_args(parser):
-    parser.add_argument('--url', help='Testing Farm API URL')
-    parser.add_argument('--token', help='Testing Farm API auth token')
+    parser.add_argument('--url', help="Testing Farm API URL")
+    parser.add_argument('--token', help="Testing Farm API auth token")
     cmds = parser.add_subparsers(
         dest='_cmd', help="TF helper to run", metavar='<cmd>', required=True,
     )
 
+    cmd = cmds.add_parser(
+        'whoami',
+        help="print out details about active TF token",
+    )
+    cmd = cmds.add_parser(
+        'about',
+        help="print out details about TF instance (url)",
+    )
+
     cmd = cmds.add_parser(
         'composes',
         help="list all composes available on a given ranch",
@@ -122,6 +165,12 @@ def parse_args(parser):
     )
     cmd.add_argument('request_id', help="Testing Farm request UUID")
 
+    cmd = cmds.add_parser(
+        'cancel',
+        help="cancel a Testing Farm request",
+    )
+    cmd.add_argument('request_id', help="Testing Farm request UUID")
+
     cmd = cmds.add_parser(
         'search-requests', aliases=('sr',),
         help="return a list of requests matching the criteria",
@@ -140,6 +189,7 @@ def parse_args(parser):
     cmd.add_argument('--arch', '-a', help="system HW architecture", default='x86_64')
     cmd.add_argument('--timeout', '-t', help="pipeline timeout (in minutes)", type=int, default=60)
     cmd.add_argument('--ssh-key', help="path to a ssh private key file like 'id_rsa'")
+    cmd.add_argument('--hvm', help="request a HVM virtualization capable HW", action='store_true')
 
     cmd = cmds.add_parser(
         'watch-pipeline', aliases=('wp',),
@@ -149,10 +199,16 @@ def parse_args(parser):
 
 
 def main(args):
-    if args._cmd == 'composes':
+    if args._cmd == 'whoami':
+        whoami(args)
+    elif args._cmd == 'about':
+        about(args)
+    elif args._cmd == 'composes':
         composes(args)
     elif args._cmd in ('get-request', 'gr'):
         get_request(args)
+    elif args._cmd == 'cancel':
+        cancel(args)
     elif args._cmd in ('search-requests', 'sr'):
         search_requests(args)
     elif args._cmd == 'reserve':

{atex-0.2 → atex-0.4}/atex/minitmt/__init__.py

@@ -1,5 +1,8 @@
+import os
 import random
 
+from pathlib import Path
+
 # TODO: TMT_PLAN_ENVIRONMENT_FILE
 
 # TODO: install rsync on the guest as part of setup
@@ -84,6 +87,9 @@ class Executor:
         # TODO: install rsync
         pass
 
+    def run_script(self, script, duration=None, shell='/bin/bash', **kwargs):
+        self.conn.ssh(shell, input=script.encode())
+
     # run one test via ssh and parse its results on-the-fly,
     # write out logs
     def run_test(self, fmf_test, reporter):

{atex-0.2 → atex-0.4}/atex/provision/libvirt/__init__.py

@@ -1,6 +1,7 @@
 from .. import Provisioner as _Provisioner
 from ... import util, ssh
 
+
 class LibvirtProvisioner(_Provisioner):
     number = 123
 

{atex-0.2 → atex-0.4}/atex/ssh.py

@@ -256,7 +256,7 @@ class SSHConn:
         unified_options['RemoteCommand'] = _shell_cmd((cmd, *args), sudo=sudo)
         unified_options['ControlPath'] = self.tmpdir / 'control.sock'
         return func(
-            _options_to_cli(unified_options, password=self.password),
+            _options_to_ssh(unified_options, password=self.password),
             skip_frames=1,
             text=text,
             **run_kwargs,
@@ -300,7 +300,7 @@ def ssh(
     unified_options['RemoteCommand'] = _shell_cmd((cmd, *args), sudo=sudo)
     unified_options.update(options)
     return func(
-        _options_to_cli(unified_options, password=password),
+        _options_to_ssh(unified_options, password=password),
         skip_frames=1,
         text=text,
         **run_kwargs,

{atex-0.2 → atex-0.4}/atex/testingfarm.py

@@ -11,7 +11,6 @@ from pathlib import Path
 
 from . import util
 
-#from pprint import pprint as pp
 import json
 import urllib3
 
@@ -22,14 +21,10 @@ API_QUERY_DELAY = 10
 
 RESERVE_TASK = {
     'fmf': {
-#        'url': 'https://github.com/RHSecurityCompliance/atex',  # TODO
-#        'ref': 'main',
-#        'path': 'fmf_tests',
-#        'name': "/reserve",
-        'url': 'https://github.com/comps/tmt-experiments-public',
-        'ref': 'master',
-        'path': '.',
-        'test_name': '/reserve',
+        'url': 'https://github.com/RHSecurityCompliance/atex',
+        'ref': 'main',
+        'path': 'fmf_tests',
+        'name': "/reserve",
     },
 }
 
@@ -44,7 +39,9 @@ _http = urllib3.PoolManager(maxsize=3, block=True)
 
 
 class TestingFarmError(Exception):
-    pass
+    def __init__(self, message, reply=None):
+        super().__init__(message)
+        self.reply = reply
 
 
 class APIError(TestingFarmError):
@@ -55,7 +52,6 @@ class BadHTTPError(TestingFarmError):
     pass
 
 
-# TODO: __init__ and __str__ so we pass just request ID, not a full message
 class GoneAwayError(TestingFarmError):
     pass
 
@@ -90,18 +86,21 @@ class TestingFarmAPI:
         reply = _http.request(method, url, *args, headers=headers, preload_content=False, **kwargs)
 
         if reply.status != 200 and not reply.data:
-            raise APIError(f"got HTTP {reply.status} on {method} {url}")
+            raise APIError(f"got HTTP {reply.status} on {method} {url}", reply)
 
         if reply.headers.get('Content-Type') != 'application/json':
-            raise BadHTTPError(f"HTTP {reply.status} on {method} {url} is not application/json")
+            raise BadHTTPError(
+                f"HTTP {reply.status} on {method} {url} is not application/json",
+                reply,
+            )
 
         try:
             decoded = reply.json()
         except json.decoder.JSONDecodeError:
-            raise BadHTTPError(f"failed to decode JSON for {method} {url}: {reply.data}")
+            raise BadHTTPError(f"failed to decode JSON for {method} {url}: {reply.data}", reply)
 
         if reply.status != 200:
-            raise APIError(f"got HTTP {reply.status} on {method} {url}: {decoded}")
+            raise APIError(f"got HTTP {reply.status} on {method} {url}: {decoded}", reply)
 
         return decoded
 
@@ -230,7 +229,8 @@ class Request:
 
     def assert_alive(self):
         if not self.alive():
-            raise GoneAwayError(f"request {self.data['id']} not alive anymore")
+            state = self.data['state']
+            raise GoneAwayError(f"request {self.data['id']} not alive anymore, entered: {state}")
 
     def wait_for_state(self, state):
         if 'state' not in self.data:
@@ -268,12 +268,28 @@ class PipelineLogStreamer:
         while True:
             self.request.wait_for_state('running')
 
-            if 'run' in self.request and 'artifacts' in self.request['run']:
-                if artifacts := self.request['run']['artifacts']:
-                    return f'{artifacts}/pipeline.log'
+            try:
+                if 'run' not in self.request or 'artifacts' not in self.request['run']:
+                    continue
+
+                artifacts = self.request['run']['artifacts']
+                if not artifacts:
+                    continue
 
-            time.sleep(API_QUERY_DELAY)
-            self.request.update()
+                log = f'{artifacts}/pipeline.log'
+                reply = _http.request('HEAD', log)
+                # TF has a race condition of adding the .log entry without it being created
+                if reply.status == 404:
+                    util.debug(f"got 404 for {log}, retrying")
+                    continue
+                elif reply.status != 200:
+                    raise APIError(f"got HTTP {reply.status} on HEAD {log}", reply)
+
+                return log
+
+            finally:
+                time.sleep(API_QUERY_DELAY)
+                self.request.update()
 
     def __iter__(self):
         url = self._wait_for_entry()
@@ -282,30 +298,30 @@ class PipelineLogStreamer:
         while True:
             self.request.assert_alive()
 
-            headers = {'Range': f'bytes={bytes_read}-'}
-            # load all returned data via .decode() rather than streaming it
-            # in chunks, because we don't want to leave the connection open
-            # (blocking others) while the user code runs between __next__ calls
-            reply = _http.request('GET', url, headers=headers)
-
-            # 416=Range Not Satisfiable, typically meaning "no new data to send"
-            if reply.status == 416:
-                time.sleep(API_QUERY_DELAY)
-                self.request.update()
-                continue
-            # 200=OK or 206=Partial Content
-            elif reply.status not in (200,206):
-                raise BadHTTPError(f"got {reply.status} when trying to GET {url}")
+            try:
+                headers = {'Range': f'bytes={bytes_read}-'}
+                # load all returned data via .decode() rather than streaming it
+                # in chunks, because we don't want to leave the connection open
+                # (blocking others) while the user code runs between __next__ calls
+                reply = _http.request('GET', url, headers=headers)
+
+                # 416=Range Not Satisfiable, typically meaning "no new data to send"
+                if reply.status == 416:
+                    continue
+                # 200=OK or 206=Partial Content
+                elif reply.status not in (200,206):
+                    raise BadHTTPError(f"got {reply.status} when trying to GET {url}", reply)
 
-            bytes_read += len(reply.data)
-            buffer += reply.data.decode(errors='ignore')
+                bytes_read += len(reply.data)
+                buffer += reply.data.decode(errors='ignore')
 
-            while (index := buffer.find('\n')) != -1:
-                yield buffer[:index]
-                buffer = buffer[index+1:]
+                while (index := buffer.find('\n')) != -1:
+                    yield buffer[:index]
+                    buffer = buffer[index+1:]
 
-            time.sleep(API_QUERY_DELAY)
-            self.request.update()
+            finally:
+                time.sleep(API_QUERY_DELAY)
+                self.request.update()
 
 
 class Reserve:
@@ -456,13 +472,6 @@ class Reserve:
             self.request.submit(spec)
             util.debug(f"submitted request:\n{textwrap.indent(str(self.request), '    ')}")
 
-            # wait for the request to become running
-            while self.request['state'] != 'running':
-                time.sleep(API_QUERY_DELAY)
-                self.request.update()
-                if self.request['state'] in END_STATES:
-                    raise GoneAwayError(f"request {self.request['id']} not alive anymore")
-
             # wait for user/host to ssh to
             ssh_user = ssh_host = None
             for line in PipelineLogStreamer(self.request):
@@ -487,10 +496,10 @@ class Reserve:
                 f'{ssh_user}@{ssh_host}', 'exit 123',
             ]
             while True:
+                # wait for API_QUERY_DELAY between ssh retries, seems like GEFN sleep time
                 time.sleep(API_QUERY_DELAY)
                 self.request.update()
-                if self.request['state'] in END_STATES:
-                    raise GoneAwayError(f"request {self.request['id']} not alive anymore")
+                self.request.assert_alive()
 
                 proc = util.subprocess_run(
                     ssh_attempt_cmd,

atex-0.4/fmf_tests/reserve/main.fmf

@@ -0,0 +1,2 @@
+test: ./test.sh
+duration: 1000h

atex-0.4/fmf_tests/reserve/test.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+set -e -x
+
+# remove useless daemons to free up RAM a bit
+dnf remove -y rng-tools irqbalance
+
+# clean up packages from extra repos, restoring original vanilla OS (sorta)
+rm -v -f \
+    /etc/yum.repos.d/{tag-repository,*beakerlib*,rcmtools}.repo \
+    /etc/yum.repos.d/beaker-{client,harness,tasks}.repo
+
+function list_foreign_rpms {
+    dnf list --installed \
+    | grep -e @koji-override -e @testing-farm -e @epel -e @copr: -e @rcmtools \
+    | sed 's/ .*//'
+}
+rpms=$(list_foreign_rpms)
+[[ $rpms ]] && dnf downgrade -y --skip-broken $rpms
+rpms=$(list_foreign_rpms)
+[[ $rpms ]] && dnf remove -y --noautoremove $rpms
+dnf clean all
+
+# install SSH key
+if [[ $RESERVE_SSH_PUBKEY ]]; then
+    mkdir -p ~/.ssh
+    chmod 0700 ~/.ssh
+    echo "$RESERVE_SSH_PUBKEY" >> ~/.ssh/authorized_keys
+    chmod 0600 ~/.ssh/authorized_keys
+else
+    echo "RESERVE_SSH_PUBKEY env var not defined" >&2
+    exit 1
+fi
+
+# wait forever
+sleep inf

{atex-0.2 → atex-0.4}/pyproject.toml

@@ -5,7 +5,8 @@ build-backend = "hatchling.build"
 [project]
 name = "atex"
 description = "Ad-hoc Test EXecutor"
-version = "0.2"
+readme = "README.md"
+version = "0.4"
 license = "GPL-3.0-or-later"
 requires-python = ">= 3.9"
 dependencies = [

atex-0.2/PKG-INFO

@@ -1,13 +0,0 @@
-Metadata-Version: 2.4
-Name: atex
-Version: 0.2
-Summary: Ad-hoc Test EXecutor
-Project-URL: Homepage, https://github.com/RHSecurityCompliance/atex
-License-Expression: GPL-3.0-or-later
-License-File: COPYING.txt
-Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Software Development :: Testing
-Requires-Python: >=3.9
-Requires-Dist: fmf>=1.6
-Requires-Dist: urllib3<3,>=2