atex 0.7__tar.gz → 0.8__tar.gz

{atex-0.7 → atex-0.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atex
-Version: 0.7
+Version: 0.8
 Summary: Ad-hoc Test EXecutor
 Project-URL: Homepage, https://github.com/RHSecurityCompliance/atex
 License-Expression: GPL-3.0-or-later
@@ -8,7 +8,7 @@ License-File: COPYING.txt
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Software Development :: Testing
-Requires-Python: >=3.9
+Requires-Python: >=3.11
 Requires-Dist: fmf>=1.6
 Requires-Dist: urllib3<3,>=2
 Description-Content-Type: text/markdown
@@ -45,8 +45,103 @@ BREAK. DO NOT USE IT (for now).
 Unless specified otherwise, any content within this repository is distributed
 under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
 
+## Parallelism and cleanup
+
+There are effectively 3 methods of running things in parallel in Python:
+
+- `threading.Thread` (and related `concurrent.futures` classes)
+- `multiprocessing.Process` (and related `concurrent.futures` classes)
+- `asyncio`
+
+and there is no clear winner (in terms of cleanup on `SIGTERM` or Ctrl-C):
+
+- `Thread` has signal handlers only in the main thread and is unable to
+  interrupt any running threads without super ugly workarounds like `sleep(1)`
+  in every thread, checking some "pls exit" variable
+- `Process` is too heavyweight and makes sharing native Python objects hard,
+  but it does handle signals in each process individually
+- `asyncio` handles interrupting perfectly (every `try`/`except`/`finally`
+  completes just fine, `KeyboardInterrupt` is raised in every async context),
+  but async python is still (3.14) too weird and unsupported
+  - `asyncio` effectively re-implements `subprocess` with a slightly different
+    API, same with `asyncio.Transport` and derivatives reimplementing `socket`
+  - 3rd party libraries like `requests` or `urllib3` don't support it, one needs
+    to resort to spawning these in separate threads anyway
+  - same with `os.*` functions and syscalls
+  - every thing exposed via API needs to have 2 copies - async and non-async,
+    making it unbearable
+  - other stdlib bugs, ie. "large" reads returning BlockingIOError sometimes
+
+The approach chosen by this project was to use `threading.Thread`, and
+implement thread safety for classes and their functions that need it.  
+For example:
+
+```python
+class MachineReserver:
+    def __init__(self):
+        self.lock = threading.RLock()
+        self.job = None
+        self.proc = None
+
+    def reserve(self, ...):
+        try:
+            ...
+            job = schedule_new_job_on_external_service()
+            with self.lock:
+                self.job = job
+            ...
+            while not reserved(self.job):
+                time.sleep(60)
+            ...
+            with self.lock:
+                self.proc = subprocess.Popen(["ssh", f"{user}@{host}", ...)
+            ...
+            return machine
+        except Exception:
+            self.abort()
+            raise
+
+    def abort(self):
+        with self.lock:
+            if self.job:
+                cancel_external_service(self.job)
+                self.job = None
+            if self.proc:
+                self.proc.kill()
+                self.proc = None
+```
+
+Here, it is expected for `.reserve()` to be called in a long-running thread that
+provisions a new machine on some external service, waits for it to be installed
+and reserved, connects an ssh session to it and returns it back.
+
+But equally, `.abort()` can be called from an external thread and clean up any
+non-pythonic resources (external jobs, processes, temporary files, etc.) at
+which point **we don't care what happens to .reserve()**, it will probably fail
+with some exception, but doesn't do any harm.
+
+Here is where `daemon=True` threads come in handy - we can simply call `.abort()`
+from a `KeyboardInterrupt` (or `SIGTERM`) handle in the main thread, and just
+exit, automatically killing any leftover threads that are uselessly sleeping.  
+(Realistically, we might want to spawn new threads to run many `.abort()`s in
+parallel, but the main thread can wait for those just fine.)
+
+It is not perfect, but it's probably the best Python can do.
+
+Note that races can still occur between a resource being reserved and written
+to `self.*` for `.abort()` to free, so resource de-allocation is not 100%
+guaranteed, but single-threaded interrupting has the same issue.  
+Do have fallbacks (ie. max reserve times on the external service).
+
+Also note that `.reserve()` and `.abort()` could be also called by a context
+manager as `__enter__` and `__exit__`, ie. by a non-threaded caller (running
+everything in the main thread).
+
+
 ## Unsorted notes
 
+TODO: codestyle from contest
+
 ```
 - this is not tmt, the goal is to make a python toolbox *for* making runcontest
   style tools easily, not to replace those tools with tmt-style CLI syntax

{atex-0.7 → atex-0.8}/README.md

@@ -30,8 +30,103 @@ BREAK. DO NOT USE IT (for now).
 Unless specified otherwise, any content within this repository is distributed
 under the GNU GPLv3 license, see the [COPYING.txt](COPYING.txt) file for more.
 
+## Parallelism and cleanup
+
+There are effectively 3 methods of running things in parallel in Python:
+
+- `threading.Thread` (and related `concurrent.futures` classes)
+- `multiprocessing.Process` (and related `concurrent.futures` classes)
+- `asyncio`
+
+and there is no clear winner (in terms of cleanup on `SIGTERM` or Ctrl-C):
+
+- `Thread` has signal handlers only in the main thread and is unable to
+  interrupt any running threads without super ugly workarounds like `sleep(1)`
+  in every thread, checking some "pls exit" variable
+- `Process` is too heavyweight and makes sharing native Python objects hard,
+  but it does handle signals in each process individually
+- `asyncio` handles interrupting perfectly (every `try`/`except`/`finally`
+  completes just fine, `KeyboardInterrupt` is raised in every async context),
+  but async python is still (3.14) too weird and unsupported
+  - `asyncio` effectively re-implements `subprocess` with a slightly different
+    API, same with `asyncio.Transport` and derivatives reimplementing `socket`
+  - 3rd party libraries like `requests` or `urllib3` don't support it, one needs
+    to resort to spawning these in separate threads anyway
+  - same with `os.*` functions and syscalls
+  - every thing exposed via API needs to have 2 copies - async and non-async,
+    making it unbearable
+  - other stdlib bugs, ie. "large" reads returning BlockingIOError sometimes
+
+The approach chosen by this project was to use `threading.Thread`, and
+implement thread safety for classes and their functions that need it.  
+For example:
+
+```python
+class MachineReserver:
+    def __init__(self):
+        self.lock = threading.RLock()
+        self.job = None
+        self.proc = None
+
+    def reserve(self, ...):
+        try:
+            ...
+            job = schedule_new_job_on_external_service()
+            with self.lock:
+                self.job = job
+            ...
+            while not reserved(self.job):
+                time.sleep(60)
+            ...
+            with self.lock:
+                self.proc = subprocess.Popen(["ssh", f"{user}@{host}", ...)
+            ...
+            return machine
+        except Exception:
+            self.abort()
+            raise
+
+    def abort(self):
+        with self.lock:
+            if self.job:
+                cancel_external_service(self.job)
+                self.job = None
+            if self.proc:
+                self.proc.kill()
+                self.proc = None
+```
+
+Here, it is expected for `.reserve()` to be called in a long-running thread that
+provisions a new machine on some external service, waits for it to be installed
+and reserved, connects an ssh session to it and returns it back.
+
+But equally, `.abort()` can be called from an external thread and clean up any
+non-pythonic resources (external jobs, processes, temporary files, etc.) at
+which point **we don't care what happens to .reserve()**, it will probably fail
+with some exception, but doesn't do any harm.
+
+Here is where `daemon=True` threads come in handy - we can simply call `.abort()`
+from a `KeyboardInterrupt` (or `SIGTERM`) handle in the main thread, and just
+exit, automatically killing any leftover threads that are uselessly sleeping.  
+(Realistically, we might want to spawn new threads to run many `.abort()`s in
+parallel, but the main thread can wait for those just fine.)
+
+It is not perfect, but it's probably the best Python can do.
+
+Note that races can still occur between a resource being reserved and written
+to `self.*` for `.abort()` to free, so resource de-allocation is not 100%
+guaranteed, but single-threaded interrupting has the same issue.  
+Do have fallbacks (ie. max reserve times on the external service).
+
+Also note that `.reserve()` and `.abort()` could be also called by a context
+manager as `__enter__` and `__exit__`, ie. by a non-threaded caller (running
+everything in the main thread).
+
+
 ## Unsorted notes
 
+TODO: codestyle from contest
+
 ```
 - this is not tmt, the goal is to make a python toolbox *for* making runcontest
   style tools easily, not to replace those tools with tmt-style CLI syntax

{atex-0.7 → atex-0.8}/TODO

@@ -1,4 +1,50 @@
-- concept of a RemoteSlot for Orchestrator ; basically, Orchastrator can
+- Orchestrator
+  - about platforms
+    - platform in Orchestrator is distro+arch when applied to a Provisioner class
+      - it is specific to that Provisioner, ie. CentOS-Stream-9 @ x86_64 on a TF Provisioner
+  - getting a new machine
+    - get a Remote from a Provisioner
+    - upload tests to it
+    - instantiate Executor for it
+      - run plan setup
+    - put it in the pool of Remotes to run tests on
+  - removing machines
+    - when all tests for a given platform have finished successfully
+      (all reruns also concluded)
+  - maintaining machines
+    - maintain an dict() of set()s of Remotes, indexed by platform (namedtuple?)
+  - instantiate a ResultAggregator for each platform
+    - probably a dict(), indexed by platform (namedtuple?)
+  - to-be-run tests
+    - discovered from fmf, one FMFTests instance for each platform
+  - algorithm?
+    - get platforms from user (ie. CentOS-Stream-9@x86_64@TestingFarmProvisioner)
+      - these MUST be distro/arch/provisioner, not random strings)
+    - discover tests using context built from the platform (distro/arch),
+      index the FMFTests in a dict() by platform name (or namedtuple?)
+    - instantiate a Provisioner for each platform, index them in a dict()
+      - may need some translation, ie. "latest-9" to a specific compose,
+        probably done by Provisioner functions
+    - while True
+      - for each provisioner instance, try to get a Remote
+      - if we get a Remote:
+        - run setup on it (see above), possibly in a separate thread
+      - go over Remote instances in some "Remotes waiting to be set up" list()
+        - if an instance is ready, find a to-be-run test, start executing it
+          and put the Remote into another "Remotes running tests" list()
+      - go over Remotes in the "Remotes running tests" list()
+        - if a Remote has finished, 
+
+- CLI
+  - atex orch \
+        platform -n 9.6@x86_64 -c distro=rhel-9.6 -c arch=x86_64 -p "TestingFarmProvisioner:RHEL-9.6.0-Nightly,x86_64" \
+        platform ... \
+        tests --repo tests_repo -p /plans/someplan -t /some/test -t /another/test -e SOME=ENV \
+        content --repo content_repo \
+        ...
+
+
+- concept of a RemoteSlot for Orchestrator ; basically, Orchestrator can
   instantiate Provisioner instances in two ways:
   - directly from given via pre-configured Provisioner classes (factories)
   - indirectly from a list of RemoteSlot instances (classes?)

atex-0.8/aggrtest.py

@@ -0,0 +1,74 @@
+#!/usr/bin/python3
+
+import sys
+import logging
+from pathlib import Path
+import shutil
+import tempfile
+import concurrent.futures
+
+#from atex.provision.testingfarm import TestingFarmProvisioner
+
+from atex import executor, connection, fmf, orchestrator
+
+
+logging.basicConfig(
+    level=logging.DEBUG,
+    stream=sys.stderr,
+    format="%(asctime)s %(name)s: %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+fmf_tests = fmf.FMFTests(
+    "/home/jjaburek/gitit/tmt-experiments",
+    "/plans/friday-demo",
+    context=None,
+)
+
+shutil.rmtree("/tmp/testme")
+Path("/tmp/testme").mkdir()
+
+ssh_options = {
+    "User": "root",
+    "Hostname": "18.119.100.84",
+    "IdentityFile": "/tmp/tmpbq4yl7es/key_rsa",
+}
+
+print("\n\n------------------\n\n")
+
+with connection.ssh.ManagedSSHConn(options=ssh_options) as conn:
+    conn.cmd(["mkdir", "/var/myatex"])
+    with executor.Executor(fmf_tests, conn, state_dir="/var/myatex") as ex:
+        ex.upload_tests("/home/jjaburek/gitit/tmt-experiments")
+        ex.setup_plan()
+
+aggr = orchestrator.CSVAggregator("/tmp/csv_file", "/tmp/storage_dir")
+aggr.open()
+print(f"\n\n====== {aggr.csv_writer} =====\n\n")
+
+def run_one(num):
+    with connection.ssh.ManagedSSHConn(options=ssh_options) as conn:
+        with executor.Executor(fmf_tests, conn, state_dir="/var/myatex") as ex:
+            for test_name in fmf_tests.tests:
+                safe_test_name = test_name.strip("/").replace("/","-")
+                # TODO: actually delete them if test passed (or leave them if some DEBUG was set)
+                tmpdir = tempfile.TemporaryDirectory(prefix=f"{safe_test_name}-", delete=False, dir="/tmp/testme")
+                files_dir = Path(tmpdir.name) / "files"
+                json_file = Path(tmpdir.name) / "json"
+                ex.run_test(test_name, json_file, files_dir)
+                aggr.ingest(f"platform-{num}", test_name, json_file, files_dir)
+
+print("\n\n------------------\n\n")
+
+run_one(1)
+n = 2
+
+with concurrent.futures.ThreadPoolExecutor(max_workers=9) as ex:
+    for _ in range(9):
+        ex.submit(run_one, n)
+        n += 1
+
+aggr.close()
+
+with connection.ssh.ManagedSSHConn(options=ssh_options) as conn:
+    conn.cmd(["rm", "-rf", "/var/myatex"])

atex-0.8/atex/cli/fmf.py

@@ -0,0 +1,93 @@
+import sys
+import pprint
+
+from .. import fmf
+
+
+def _fatal(msg):
+    print(msg, file=sys.stderr)
+    sys.exit(1)
+
+
+def _get_context(args):
+    context = {}
+    if args.context:
+        for c in args.context:
+            key, value = c.split("=", 1)
+            context[key] = value
+    return context or None
+
+
+def discover(args):
+    result = fmf.FMFTests(args.root, args.plan, context=_get_context(args))
+    for name in result.tests:
+        print(name)
+
+
+def show(args):
+    result = fmf.FMFTests(args.root, args.plan, context=_get_context(args))
+    if tests := list(result.match(args.test)):
+        for test in tests:
+            print(f"\n--- {test.name} ---")
+            pprint.pprint(test.data)
+    else:
+        _fatal(f"Not reachable via {args.plan} discovery: {args.test}")
+
+
+def prepare(args):
+    result = fmf.FMFTests(args.root, args.plan, context=_get_context(args))
+    print("--- fmf root ---")
+    print(str(result.root))
+    print("--- prepare packages ---")
+    print("\n".join(result.prepare_pkgs))
+    print("--- plan environment ---")
+    print("\n".join("{k}={v}" for k,v in result.plan_env))
+    for script in result.prepare_scripts:
+        print("--- prepare script ---")
+        print(script)
+        print("----------------------")
+
+
+def parse_args(parser):
+    parser.add_argument("--root", help="path to directory with fmf tests", default=".")
+    parser.add_argument("--context", "-c", help="tmt style key=value context", action="append")
+    cmds = parser.add_subparsers(
+        dest="_cmd", help="executor feature", metavar="<cmd>", required=True,
+    )
+
+    cmd = cmds.add_parser(
+        "discover", aliases=("di",),
+        help="list tests, post-processed by tmt plans",
+    )
+    cmd.add_argument("plan", help="tmt plan to use for discovery")
+
+    cmd = cmds.add_parser(
+        "show",
+        help="show fmf data of a test",
+    )
+    cmd.add_argument("plan", help="tmt plan to use for discovery")
+    cmd.add_argument("test", help="fmf style test regex")
+
+    cmd = cmds.add_parser(
+        "prepare",
+        help="show prepare-related FMFTests details",
+    )
+    cmd.add_argument("plan", help="tmt plan to parse")
+
+
+def main(args):
+    if args._cmd in ("discover", "di"):
+        discover(args)
+    elif args._cmd == "show":
+        show(args)
+    elif args._cmd == "prepare":
+        prepare(args)
+    else:
+        raise RuntimeError(f"unknown args: {args}")
+
+
+CLI_SPEC = {
+    "help": "simple CLI interface to atex.fmf",
+    "args": parse_args,
+    "main": main,
+}

{atex-0.7 → atex-0.8}/atex/cli/testingfarm.py

@@ -1,4 +1,5 @@
 import sys
+import json
 import pprint
 
 from .. import util
@@ -49,6 +50,8 @@ def search_requests(args):
     reply = api.search_requests(
         state=args.state,
         mine=not args.all,
+        user_id=args.user_id,
+        token_id=args.token_id,
         ranch=args.ranch,
         created_before=args.before,
         created_after=args.after,
@@ -56,20 +59,24 @@ def search_requests(args):
     if not reply:
         return
 
-    for req in sorted(reply, key=lambda x: x["created"]):
-        req_id = req["id"]
-        created = req["created"].partition(".")[0]
+    if args.json:
+        for req in sorted(reply, key=lambda x: x["created"]):
+            print(json.dumps(req))
+    else:
+        for req in sorted(reply, key=lambda x: x["created"]):
+            req_id = req["id"]
+            created = req["created"].partition(".")[0]
 
-        envs = []
-        for env in req["environments_requested"]:
-            if "os" in env and env["os"] and "compose" in env["os"]:
-                compose = env["os"]["compose"]
-                arch = env["arch"]
-                if compose and arch:
-                    envs.append(f"{compose}@{arch}")
-        envs_str = ", ".join(envs)
+            envs = []
+            for env in req["environments_requested"]:
+                if "os" in env and env["os"] and "compose" in env["os"]:
+                    compose = env["os"]["compose"]
+                    arch = env["arch"]
+                    if compose and arch:
+                        envs.append(f"{compose}@{arch}")
+            envs_str = ", ".join(envs)
 
-        print(f"{created} {req_id} : {envs_str}")
+            print(f"{created} {req_id} : {envs_str}")
 
 
 def reserve(args):
@@ -177,9 +184,12 @@ def parse_args(parser):
     )
     cmd.add_argument("--state", help="request state (running, etc.)", required=True)
     cmd.add_argument("--all", help="all requests, not just owned by token", action="store_true")
-    cmd.add_argument("--ranch", help="Testing Farm ranch")
+    cmd.add_argument("--ranch", help="Testing Farm ranch (detected from token)")
+    cmd.add_argument("--user-id", help="'user_id' request field (detected from token)")
+    cmd.add_argument("--token-id", help="'token_id' request field (detected from token)")
     cmd.add_argument("--before", help="only requests created before ISO8601")
     cmd.add_argument("--after", help="only requests created after ISO8601")
+    cmd.add_argument("--json", help="full details, one request per line", action="store_true")
 
     cmd = cmds.add_parser(
         "reserve",

{atex-0.7 → atex-0.8}/atex/connection/__init__.py

@@ -65,14 +65,6 @@ class Connection:
         """
         raise NotImplementedError(f"'disconnect' not implemented for {self.__class__.__name__}")
 
-    # TODO: is this needed? .. we probably want Remote.alive() instead
-    #def alive(self):
-    #    """
-    #    Return True if the connection was established and is active,
-    #    False otherwise.
-    #    """
-    #    raise NotImplementedError(f"'alive' not implemented for {self.__class__.__name__}")
-
     def cmd(self, command, func=_util.subprocess_run, **func_args):
         """
         Execute a single command on the remote, using subprocess-like semantics.

{atex-0.7 → atex-0.8}/atex/connection/ssh.py

@@ -68,7 +68,7 @@ def _shell_cmd(command, sudo=None):
     """
     Make a command line for running 'command' on the target system.
     """
-    quoted_args = (shlex.quote(arg) for arg in command)
+    quoted_args = (shlex.quote(str(arg)) for arg in command)
     if sudo:
         return " ".join((
             "exec", "sudo", "--no-update", "--non-interactive", "--user", sudo, "--", *quoted_args,
@@ -167,15 +167,12 @@ class StatelessSSHConn(Connection):
         Optional, .cmd() and .rsync() work without it, but it is provided here
         for compatibility with the Connection API.
         """
-        # TODO: just wait until .cmd(['true']) starts responding
+        # TODO: just wait until .cmd(['true']) starts responding ?
         pass
 
     def disconnect(self):
         pass
 
-#    def alive(self):
-#        return True
-
     # have options as kwarg to be compatible with other functions here
     def cmd(self, command, options=None, func=util.subprocess_run, **func_args):
         unified_options = self.options.copy()
@@ -231,7 +228,7 @@ class ManagedSSHConn(Connection):
     to manage this complexity.
     """
 
-    # TODO: thread safety and locking via self.lock
+    # TODO: thread safety and locking via self.lock ?
 
     def __init__(self, options, *, password=None, sudo=None):
         """
@@ -251,12 +248,6 @@ class ManagedSSHConn(Connection):
         self._tmpdir = None
         self._master_proc = None
 
-#    def __copy__(self):
-#        return type(self)(self.options, password=self.password)
-#
-#    def copy(self):
-#        return self.__copy__()
-
     def assert_master(self):
         proc = self._master_proc
         if not proc:
@@ -272,13 +263,6 @@ class ManagedSSHConn(Connection):
                 f"SSH ControlMaster on {self._tmpdir} exited with {code}{out}",
             )
 
-#    def alive(self):
-#        try:
-#            self.assert_master()
-#            return True
-#        except (NotConnectedError, DisconnectedError):
-#            return False
-
     def disconnect(self):
         proc = self._master_proc
         if not proc:

{atex-0.7/atex/minitmt → atex-0.8/atex/executor}/README.md

@@ -1,4 +1,4 @@
-# Minitmt
+# Executor
 
 This is a minimalistic re-implementation of some of the features of
 [tmt](https://github.com/teemtee/tmt), without re-inventing the test metadata
@@ -60,9 +60,9 @@ pieces of fmf metadata, than to deal with all of the above.
 
 ## Compatibility
 
-Minitmt is designed to be mostly-compatible with tmt in most simple use cases,
-the idea is that you should be able to write tests that **work with both**,
-easily.
+This implementation is designed to be mostly-compatible with tmt in most simple
+use cases, the idea is that you should be able to write tests that **work with
+both**, easily.
 
 Our main problem with the ecosystem around tmt is that it is heavily
 Beakerlib-inspired, with tools relying on a small subset of tmt functionality
@@ -76,10 +76,9 @@ So the goal here is to write tests that
   - having no additional logs, letting tmt use `output.txt` as test output,
     renamed to `testout.log` by Testing Farm
   - not trying to be fancy
-- run under minitmt in a more "wild" mode, without those limitations
+- run under atex in a more "wild" mode, without those limitations
   - tens of millions of results
   - logs with full paths
-  - cross-test result reporting
   - etc.
 
 Hopefully running well under Testing Farm / OSCI / etc., while being more
@@ -106,6 +105,8 @@ Everything supported by fmf should work, incl.
   - `exclude` support (custom `re`-based filter, not in fmf)
   - No remote git repo (aside from what fmf supports natively), no `check`,
     no `modified-only`, no `adjust-tests`, etc.
+  - Tests from multiple `discover` sections are added together, eg. any order
+    of the `discover` sections in the fmf is (currently) not honored.
 - `provision`
   - Ignored (custom provisioning logic used)
 - `prepare`
@@ -154,7 +155,7 @@ Everything supported by fmf should work, incl.
   - Ignored
 - `result`
   - Ignored, intentionally, see [RESULTS.md](RESULTS.md) below
-  - The intention is for you to be able to use **both** tmt and minitmt
+  - The intention is for you to be able to use **both** tmt and atex
     reporting if you want to, so `result` is for when you want full tmt
 - `restart`
   - Ignored, restart how many times you want until `duration`