pyallel 0.5.0__tar.gz → 0.6.0__tar.gz

{pyallel-0.5.0 → pyallel-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pyallel
-Version: 0.5.0
+Version: 0.6.0
 Summary: Run and handle the output of multiple executables in pyallel (as in parallel)
 Home-page: https://github.com/Danthewaann/pyallel
 License: MIT
@@ -23,6 +23,8 @@ Description-Content-Type: text/markdown
 
 Run and handle the output of multiple executables in `pyallel` (as in parallel)
 
+https://github.com/Danthewaann/pyallel/assets/22531177/be12efc4-439d-416d-8112-dc57cc4c291a
+
 Requires Python >=3.8
 
 Tested on Linux and MacOS only
@@ -37,17 +39,43 @@ pip install pyallel
 
 Once installed, you can run `pyallel` to see usage information, like so:
 
-```bash
-pyallel
+```
+usage: pyallel [-h] [-d] [-n] [-s] [-V] [-v] [commands ...]
+
+Run and handle the output of multiple executables in pyallel (as in parallel)
+
+positional arguments:
+  commands              list of quoted commands to run e.g "mypy ." "black ."
+
+                        can provide environment variables to each command like so:
+
+                             "MYPY_FORCE_COLOR=1 mypy ."
+
+                        command modes:
+
+                        can also provide modes to commands to do extra things:
+
+                            "tail=10 :: pytest ." <-- only output the last 10 lines, doesn't work in --no-stream mode
+
+options:
+  -h, --help            show this help message and exit
+  -d, --debug           output debug info for each command
+  -n, --non-interactive
+                        run in non-interactive mode
+  -s, --no-stream       don't stream output of each command
+  -V, --verbose         run in verbose mode
+  -v, --version         print version and exit
 ```
 
 Currently you can provide a variable number of `commands` to run to `pyallel`, like so:
 
 > [!IMPORTANT]
-> If your need to provide arguments to a command, you must surround the command and it's arguments in quotes!
+> If you need to provide arguments to a command, you must surround the command and it's arguments in quotes!
 
 ```bash
-pyallel "black --color --check --diff ." "MYPY_FORCE_COLOR=1 mypy ." "ruff check --no-fix ."
+pyallel "MYPY_FORCE_COLOR=1 mypy ." \
+        "black --check --diff ." \
+        "tail=20 :: pytest ."
 ```
 
 ## TODOs
@@ -55,15 +83,13 @@ pyallel "black --color --check --diff ." "MYPY_FORCE_COLOR=1 mypy ." "ruff check
 - [x] Allow output for all provided commands to be streamed to stdout (this will require a
       re-work of how we print command output as we currently just print output once the command
       finishes)
-- [ ] Add CI checks to run the tests and linters against Python versions > 3.8
-- [ ] Allow specific command outputs to be streamed to stdout, while all other
-      commands will only get outputted after the streamed commands have completed (such as running
-      `pytest` as a streamed command, whilst running `mypy`, `ruff` etc. as non-streamed commands).
-      This will require adding special arguments/modes to each command so they are treated
-      differently from other commands
-- [ ] Add visual examples of `pyallel` in action
+- [x] Add CI checks to run the tests and linters against Python versions > 3.8
+- [x] Add command mode arguments to support things like only tailing the last 10 lines
+      of a command whilst it is running e.g. `"tail=10 :: pytest ."`
+- [x] Add visual examples of `pyallel` in action
 - [ ] Add custom parsing of command output to support filtering for errors (like vim's
       `errorformat`)
+- [ ] Add graceful Ctrl-C interrupt handling
 - [x] Provide a way to set environment variables for each command to run with
 - [ ] Allow list of files to be provided to supply as input arguments to each command
 - [ ] Allow input to be piped into `pyallel` via stdin to supply as standard input to each

pyallel-0.6.0/README.md

@@ -0,0 +1,83 @@
+# Pyallel
+
+Run and handle the output of multiple executables in `pyallel` (as in parallel)
+
+https://github.com/Danthewaann/pyallel/assets/22531177/be12efc4-439d-416d-8112-dc57cc4c291a
+
+Requires Python >=3.8
+
+Tested on Linux and MacOS only
+
+# Quick start
+
+`pyallel` can be installed using pip:
+
+```bash
+pip install pyallel
+```
+
+Once installed, you can run `pyallel` to see usage information, like so:
+
+```
+usage: pyallel [-h] [-d] [-n] [-s] [-V] [-v] [commands ...]
+
+Run and handle the output of multiple executables in pyallel (as in parallel)
+
+positional arguments:
+  commands              list of quoted commands to run e.g "mypy ." "black ."
+
+                        can provide environment variables to each command like so:
+
+                             "MYPY_FORCE_COLOR=1 mypy ."
+
+                        command modes:
+
+                        can also provide modes to commands to do extra things:
+
+                            "tail=10 :: pytest ." <-- only output the last 10 lines, doesn't work in --no-stream mode
+
+options:
+  -h, --help            show this help message and exit
+  -d, --debug           output debug info for each command
+  -n, --non-interactive
+                        run in non-interactive mode
+  -s, --no-stream       don't stream output of each command
+  -V, --verbose         run in verbose mode
+  -v, --version         print version and exit
+```
+
+Currently you can provide a variable number of `commands` to run to `pyallel`, like so:
+
+> [!IMPORTANT]
+> If you need to provide arguments to a command, you must surround the command and it's arguments in quotes!
+
+```bash
+pyallel "MYPY_FORCE_COLOR=1 mypy ." \
+        "black --check --diff ." \
+        "tail=20 :: pytest ."
+```
+
+## TODOs
+
+- [x] Allow output for all provided commands to be streamed to stdout (this will require a
+      re-work of how we print command output as we currently just print output once the command
+      finishes)
+- [x] Add CI checks to run the tests and linters against Python versions > 3.8
+- [x] Add command mode arguments to support things like only tailing the last 10 lines
+      of a command whilst it is running e.g. `"tail=10 :: pytest ."`
+- [x] Add visual examples of `pyallel` in action
+- [ ] Add custom parsing of command output to support filtering for errors (like vim's
+      `errorformat`)
+- [ ] Add graceful Ctrl-C interrupt handling
+- [x] Provide a way to set environment variables for each command to run with
+- [ ] Allow list of files to be provided to supply as input arguments to each command
+- [ ] Allow input to be piped into `pyallel` via stdin to supply as standard input to each
+      command
+- [ ] Add custom config file for `pyallel` to read from as an alternative to providing
+      arguments via the command line
+- [ ] Add support for providing config via a `[tool.pyallel]` section in a
+      `pyproject.toml` file in the current working directory
+- [ ] Maybe allow command dependencies to be defined in a python file where commands are
+      decorated with info that details it's dependencies?
+- [x] Add test suite
+- [x] Improve error handling when parsing provided commands (check they are valid executables)

{pyallel-0.5.0 → pyallel-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pyallel"
-version = "0.5.0"
+version = "0.6.0"
 description = "Run and handle the output of multiple executables in pyallel (as in parallel)"
 authors = ["Daniel Black <danielcrblack@gmail.com>"]
 license = "MIT"

{pyallel-0.5.0 → pyallel-0.6.0}/src/pyallel/main.py

@@ -1,8 +1,10 @@
 from __future__ import annotations
 
 import sys
+import traceback
 import time
 import importlib.metadata
+from pyallel.errors import InvalidExecutableErrors
 
 from pyallel.parser import Arguments, create_parser
 from pyallel.process import ProcessGroup, format_time_taken
@@ -11,13 +13,12 @@ from pyallel import constants
 
 def main_loop(
     commands: list[str],
-    fail_fast: bool = False,
     interactive: bool = False,
     debug: bool = False,
     stream: bool = False,
 ) -> bool:
     process_group = ProcessGroup.from_commands(
-        commands, interactive=interactive, fail_fast=fail_fast, debug=debug
+        commands, interactive=interactive, debug=debug
     )
     if not stream:
         return process_group.run()
@@ -48,14 +49,16 @@ def run(*args: str) -> int:
     try:
         status = main_loop(
             parsed_args.commands,
-            parsed_args.fail_fast,
             parsed_args.interactive,
             parsed_args.debug,
             parsed_args.stream,
         )
-    except Exception as e:
+    except InvalidExecutableErrors as e:
         status = False
         message = str(e)
+    except Exception:
+        status = False
+        message = traceback.format_exc()
 
     if not status:
         if not message:

{pyallel-0.5.0 → pyallel-0.6.0}/src/pyallel/parser.py

@@ -5,7 +5,6 @@ from argparse import ArgumentParser, RawTextHelpFormatter
 
 class Arguments:
     commands: list[str]
-    fail_fast: bool
     interactive: bool
     debug: bool
     verbose: bool
@@ -20,6 +19,20 @@ class Arguments:
         return msg
 
 
+COMMANDS_HELP = """list of quoted commands to run e.g "mypy ." "black ."
+
+can provide environment variables to each command like so:
+
+     "MYPY_FORCE_COLOR=1 mypy ."
+
+command modes:
+
+can also provide modes to commands to do extra things:
+
+    "tail=10 :: pytest ." <-- only output the last 10 lines, doesn't work in --no-stream mode
+"""
+
+
 def create_parser() -> ArgumentParser:
     parser = ArgumentParser(
         prog="pyallel",
@@ -28,15 +41,13 @@ def create_parser() -> ArgumentParser:
     )
     parser.add_argument(
         "commands",
-        help='list of quoted commands to run e.g "mypy ." "black ."\n\n'
-        "can provide environment variables to each command like so:\n\n"
-        '     "MYPY_FORCE_COLOR=1 mypy ."',
+        help=COMMANDS_HELP,
         nargs="*",
     )
     parser.add_argument(
-        "-f",
-        "--fail-fast",
-        help="exit immediately when a command fails",
+        "-d",
+        "--debug",
+        help="output debug info for each command",
         action="store_true",
         default=False,
     )
@@ -48,19 +59,13 @@ def create_parser() -> ArgumentParser:
         dest="interactive",
         default=True,
     )
-    parser.add_argument(
-        "-d",
-        "--debug",
-        help="output debug info for each command",
-        action="store_true",
-        default=False,
-    )
     parser.add_argument(
         "-s",
-        "--stream",
-        help="stream output of first command",
-        action="store_true",
-        default=False,
+        "--no-stream",
+        help="don't stream output of each command",
+        action="store_false",
+        default=True,
+        dest="stream",
     )
     parser.add_argument(
         "-V",

{pyallel-0.5.0 → pyallel-0.6.0}/src/pyallel/process.py

@@ -16,7 +16,13 @@ from pyallel.errors import InvalidExecutableErrors, InvalidExecutableError
 
 
 def indent(output: str) -> str:
-    return "\n".join("    " + line for line in output.splitlines())
+    length = 200
+    indented_output = []
+    for line in output.splitlines():
+        if len(line) > length:
+            line = line[:length] + "..."
+        indented_output.append(line)
+    return "\n".join("    " + line for line in indented_output)
 
 
 def format_time_taken(time_taken: float) -> str:
@@ -48,6 +54,7 @@ def get_command_status(
     icon: str | None = None,
     passed: bool | None = None,
     debug: bool = False,
+    timer: bool = False,
 ) -> str:
     if passed is True:
         colour = constants.GREEN_BOLD
@@ -71,7 +78,7 @@ def get_command_status(
 
     output += f"{constants.NC}]{colour} {msg} "
 
-    if debug:
+    if timer:
         end = process.end
         if not process.end:
             end = time.perf_counter()
@@ -93,11 +100,11 @@ def run_process(process: Process, debug: bool = False) -> bool:
     print(f"{constants.CLEAR_LINE}{constants.CR}", end="")
 
     if process.return_code() != 0:
-        print(get_command_status(process, passed=False, debug=debug))
+        print(get_command_status(process, passed=False, debug=debug, timer=debug))
         print_command_output(process)
         return False
     else:
-        print(get_command_status(process, passed=True, debug=debug))
+        print(get_command_status(process, passed=True, debug=debug, timer=debug))
         print_command_output(process)
         return True
 
@@ -105,7 +112,6 @@ def run_process(process: Process, debug: bool = False) -> bool:
 @dataclass
 class ProcessGroup:
     processes: list[Process]
-    fail_fast: bool = False
     interactive: bool = False
     debug: bool = False
     output: dict[UUID, str] = field(default_factory=dict)
@@ -134,9 +140,9 @@ class ProcessGroup:
                     continue
 
                 completed_processes.add(process.id)
-                passed = run_process(process, debug=self.debug)
-                if self.fail_fast and not passed:
-                    return False
+                process_passed = run_process(process, debug=self.debug)
+                if not process_passed:
+                    passed = False
 
             if len(completed_processes) == len(self.processes):
                 break
@@ -160,19 +166,31 @@ class ProcessGroup:
             for i, process in enumerate(self.processes, start=1):
                 if process.poll() is not None:
                     completed_processes.add(process.id)
+                    if process.return_code() != 0:
+                        passed = False
                     output += get_command_status(
-                        process, passed=process.return_code() == 0, debug=self.debug
+                        process,
+                        passed=process.return_code() == 0,
+                        debug=self.debug,
+                        timer=self.debug,
                     )
                     output += "\n"
                 else:
                     output += get_command_status(
-                        process, icon=constants.ICONS[icon], debug=self.debug
+                        process,
+                        icon=constants.ICONS[icon],
+                        debug=self.debug,
+                        timer=self.debug,
                     )
                     output += "\n"
 
                 process_output = process.read().decode()
                 if process_output:
                     self.output[process.id] = process_output
+                    if process.tail_mode.enabled:
+                        process_output = "\n".join(
+                            process_output.splitlines()[-process.tail_mode.lines :]
+                        )
                     output += indent(process_output)
                     output += "\n"
                     if i != len(self.processes):
@@ -211,7 +229,7 @@ class ProcessGroup:
             output = ""
             for process in self.processes:
                 if running_process is None and process.id not in completed_processes:
-                    output += get_command_status(process)
+                    output += get_command_status(process, debug=self.debug)
                     output += "\n"
                     running_process = process
                 elif running_process is not process:
@@ -223,6 +241,8 @@ class ProcessGroup:
                     output += "\n"
 
                 if process.poll() is not None:
+                    if process.return_code() != 0:
+                        passed = False
                     process_output_2 = process.readline().decode()
                     while process_output_2:
                         output += indent(process_output_2)
@@ -230,7 +250,10 @@ class ProcessGroup:
                         process_output_2 = process.readline().decode()
 
                     output += get_command_status(
-                        process, passed=process.return_code() == 0, debug=self.debug
+                        process,
+                        passed=process.return_code() == 0,
+                        debug=self.debug,
+                        timer=self.debug,
                     )
                     output += "\n\n"
                     completed_processes.add(process.id)
@@ -251,7 +274,6 @@ class ProcessGroup:
         cls,
         commands: list[str],
         interactive: bool = False,
-        fail_fast: bool = False,
         debug: bool = False,
     ) -> ProcessGroup:
         processes: list[Process] = []
@@ -269,14 +291,19 @@ class ProcessGroup:
         return cls(
             processes=processes,
             interactive=interactive,
-            fail_fast=fail_fast,
             debug=debug,
         )
 
 
+@dataclass
+class TailMode:
+    enabled: bool = False
+    lines: int = 0
+
+
 @dataclass
 class Process:
-    id: UUID
+    id: UUID = field(repr=False, compare=False)
     name: str
     args: list[str]
     env: dict[str, str] = field(default_factory=dict)
@@ -287,6 +314,7 @@ class Process:
     fd_name: Path | None = None
     fd_read: BinaryIO | None = None
     fd: int | None = None
+    tail_mode: TailMode = field(default_factory=TailMode)
 
     def run(self) -> None:
         self.start = time.perf_counter()
@@ -302,8 +330,6 @@ class Process:
     def __del__(self) -> None:
         if self.fd_read:
             self.fd_read.close()
-        if self.fd_name:
-            self.fd_name.unlink(missing_ok=True)
 
     def poll(self) -> int | None:
         if self.process:
@@ -332,14 +358,19 @@ class Process:
 
     @classmethod
     def from_command(cls, command: str) -> Process:
+        tail_mode = TailMode()
         env = os.environ.copy()
         if " :: " in command:
-            command_modes, args = command.split(" :: ")
-            command_modes = command_modes.split()
-            args = args.split()
+            modes, _args = command.split(" :: ")
+            if modes:
+                for mode in modes.split():
+                    name, value = mode.split("=", maxsplit=1)
+                    if name == "tail":
+                        tail_mode.enabled = True
+                        tail_mode.lines = int(value)
+            args = _args.split()
         else:
             args = command.split()
-            command_modes = ""
 
         parsed_args: list[str] = []
         for arg in args:
@@ -353,4 +384,6 @@ class Process:
             raise InvalidExecutableError(parsed_args[0])
 
         str_args = shlex.split(" ".join(parsed_args[1:]))
-        return cls(id=uuid4(), name=parsed_args[0], args=str_args, env=env)
+        return cls(
+            id=uuid4(), name=parsed_args[0], args=str_args, env=env, tail_mode=tail_mode
+        )

pyallel-0.5.0/README.md

@@ -1,57 +0,0 @@
-# Pyallel
-
-Run and handle the output of multiple executables in `pyallel` (as in parallel)
-
-Requires Python >=3.8
-
-Tested on Linux and MacOS only
-
-# Quick start
-
-`pyallel` can be installed using pip:
-
-```bash
-pip install pyallel
-```
-
-Once installed, you can run `pyallel` to see usage information, like so:
-
-```bash
-pyallel
-```
-
-Currently you can provide a variable number of `commands` to run to `pyallel`, like so:
-
-> [!IMPORTANT]
-> If your need to provide arguments to a command, you must surround the command and it's arguments in quotes!
-
-```bash
-pyallel "black --color --check --diff ." "MYPY_FORCE_COLOR=1 mypy ." "ruff check --no-fix ."
-```
-
-## TODOs
-
-- [x] Allow output for all provided commands to be streamed to stdout (this will require a
-      re-work of how we print command output as we currently just print output once the command
-      finishes)
-- [ ] Add CI checks to run the tests and linters against Python versions > 3.8
-- [ ] Allow specific command outputs to be streamed to stdout, while all other
-      commands will only get outputted after the streamed commands have completed (such as running
-      `pytest` as a streamed command, whilst running `mypy`, `ruff` etc. as non-streamed commands).
-      This will require adding special arguments/modes to each command so they are treated
-      differently from other commands
-- [ ] Add visual examples of `pyallel` in action
-- [ ] Add custom parsing of command output to support filtering for errors (like vim's
-      `errorformat`)
-- [x] Provide a way to set environment variables for each command to run with
-- [ ] Allow list of files to be provided to supply as input arguments to each command
-- [ ] Allow input to be piped into `pyallel` via stdin to supply as standard input to each
-      command
-- [ ] Add custom config file for `pyallel` to read from as an alternative to providing
-      arguments via the command line
-- [ ] Add support for providing config via a `[tool.pyallel]` section in a
-      `pyproject.toml` file in the current working directory
-- [ ] Maybe allow command dependencies to be defined in a python file where commands are
-      decorated with info that details it's dependencies?
-- [x] Add test suite
-- [x] Improve error handling when parsing provided commands (check they are valid executables)