multiCMD 1.56__tar.gz → 1.58__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {multicmd-1.56 → multicmd-1.58}/PKG-INFO +20 -2
- {multicmd-1.56 → multicmd-1.58}/README.md +19 -1
- {multicmd-1.56 → multicmd-1.58}/multiCMD.egg-info/PKG-INFO +20 -2
- {multicmd-1.56 → multicmd-1.58}/multiCMD.egg-info/SOURCES.txt +2 -0
- {multicmd-1.56 → multicmd-1.58}/multiCMD.egg-info/entry_points.txt +0 -1
- {multicmd-1.56 → multicmd-1.58}/multiCMD.py +31 -23
- multicmd-1.58/scripts/multiCMD +6 -0
- {multicmd-1.56 → multicmd-1.58}/setup.py +3 -1
- {multicmd-1.56 → multicmd-1.58}/tests/test_async_executor.py +24 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_env.py +22 -8
- {multicmd-1.56 → multicmd-1.58}/tests/test_interrupts.py +5 -4
- multicmd-1.58/tests/test_packaging.py +26 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_process_groups.py +2 -1
- {multicmd-1.56 → multicmd-1.58}/tests/test_supervision.py +14 -4
- {multicmd-1.56 → multicmd-1.58}/LICENSE +0 -0
- {multicmd-1.56 → multicmd-1.58}/multiCMD.egg-info/dependency_links.txt +0 -0
- {multicmd-1.56 → multicmd-1.58}/multiCMD.egg-info/top_level.txt +0 -0
- {multicmd-1.56 → multicmd-1.58}/setup.cfg +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_api.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_cli.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_cli_input.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_cli_summary.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_display.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_expand_ranges.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_helpers.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_parsing.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_ping.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_pool.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_pty.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_reader.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_regressions.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_run_command.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_run_commands.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_sudo.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_task.py +0 -0
- {multicmd-1.56 → multicmd-1.58}/tests/test_windows.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: multiCMD
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.58
|
|
4
4
|
Summary: Run commands simultaneously
|
|
5
5
|
Home-page: https://github.com/yufei-pan/multiCMD
|
|
6
6
|
Author: Yufei Pan
|
|
@@ -56,7 +56,7 @@ It works both as a command-line tool and as an importable wrapper around
|
|
|
56
56
|
pip install multiCMD
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
This installs the module and three equivalent
|
|
59
|
+
This installs the module and three equivalent commands:
|
|
60
60
|
|
|
61
61
|
```bash
|
|
62
62
|
mcmd
|
|
@@ -491,6 +491,24 @@ pytest -m "not slow" # skip intentional sleep-heavy cases
|
|
|
491
491
|
|
|
492
492
|
`test_expand_ranges_perf.py` is a manual performance microbenchmark and is not collected by pytest.
|
|
493
493
|
|
|
494
|
+
## Upgrading to 1.58 from 1.57
|
|
495
|
+
|
|
496
|
+
- The `multiCMD` command is now a script file, not an entry point, so `entry_points.txt` no longer has two names that differ only in case. All three commands still work the same way, except in the venv case below. From 1.58 on, stock pip 9 on EL7 and EL8 upgrades and uninstalls multiCMD without `configparser.DuplicateOptionError`.
|
|
497
|
+
- With pip 9, the pip of EL7 and EL8 (`pip3 --version`), upgrading from 1.57 or earlier and `pip3 uninstall multiCMD` of those releases stop with `configparser.DuplicateOptionError: ... option 'multicmd' in section 'console_scripts' already exists`, and nothing changes: the installed release keeps working. Run `pip3 install -U pip` once, then upgrade or uninstall.
|
|
498
|
+
- With `--user` the same happens, except that `pip3 install --user -U multiCMD` with no version does not stop: it installs the new release, which works, and leaves the old release's metadata behind. `pip3 list` then shows the old release after `pip3 uninstall multiCMD`, and the next uninstall stops with that error. To avoid this, run `pip3 install --user -U pip` once first. If it has already happened, run that, run `pip3 uninstall multiCMD` until it says multiCMD is not installed, and install again. From 1.58 on the same upgrade leaves the metadata too, but with no error: `pip3 uninstall multiCMD` takes one run per release left.
|
|
499
|
+
- Without `--user`, with multiCMD 1.57 or earlier installed, `pip3 install -U hpcp`, `statblk` or `statbtrfs` stops the same way once a newer multiCMD is out, because pip 9 upgrades dependencies too. `pip3 install -U --no-deps hpcp` gets past it.
|
|
500
|
+
- With pip 10 or later, in a venv whose path has a space or is too long for the kernel's `#!` line, the `multiCMD` spelling does not start (the error varies: "bad interpreter", `import: command not found`), because pip gives a script file a plain `#!` line: use `multicmd`. With pip 9 no spelling starts in such a venv, as in 1.57. uv and system or `--user` installs are not affected.
|
|
501
|
+
- An embedded copy made by the generator now falls back from any failure to import the installed multiCMD, not only the three errors it named.
|
|
502
|
+
|
|
503
|
+
## Upgrading to 1.57 from 1.56
|
|
504
|
+
|
|
505
|
+
- Faster command starts on Python 3.6 - 3.10: the wrapper that puts a command in its own process group
|
|
506
|
+
imports only built-in modules, halving its CPU cost (about 7 ms to 3.5 ms per command on EL8's Python 3.6.8).
|
|
507
|
+
Nothing else about starting a command changes.
|
|
508
|
+
- Fix: `AsyncExecutor`'s `task.thread` is set by the command's thread once it runs. Before, it was set just before
|
|
509
|
+
the thread started, and a `task.thread.join()` in that gap returned at once, with the command not yet run.
|
|
510
|
+
`task.thread` from `run_command(..., wait_for_return=False)` is unchanged.
|
|
511
|
+
|
|
494
512
|
## Upgrading to 1.56 from 1.55
|
|
495
513
|
|
|
496
514
|
- Fix: a command that times out while it redraws a `\r` progress line on stderr keeps its `Timeout!`
|
|
@@ -30,7 +30,7 @@ It works both as a command-line tool and as an importable wrapper around
|
|
|
30
30
|
pip install multiCMD
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
This installs the module and three equivalent
|
|
33
|
+
This installs the module and three equivalent commands:
|
|
34
34
|
|
|
35
35
|
```bash
|
|
36
36
|
mcmd
|
|
@@ -465,6 +465,24 @@ pytest -m "not slow" # skip intentional sleep-heavy cases
|
|
|
465
465
|
|
|
466
466
|
`test_expand_ranges_perf.py` is a manual performance microbenchmark and is not collected by pytest.
|
|
467
467
|
|
|
468
|
+
## Upgrading to 1.58 from 1.57
|
|
469
|
+
|
|
470
|
+
- The `multiCMD` command is now a script file, not an entry point, so `entry_points.txt` no longer has two names that differ only in case. All three commands still work the same way, except in the venv case below. From 1.58 on, stock pip 9 on EL7 and EL8 upgrades and uninstalls multiCMD without `configparser.DuplicateOptionError`.
|
|
471
|
+
- With pip 9, the pip of EL7 and EL8 (`pip3 --version`), upgrading from 1.57 or earlier and `pip3 uninstall multiCMD` of those releases stop with `configparser.DuplicateOptionError: ... option 'multicmd' in section 'console_scripts' already exists`, and nothing changes: the installed release keeps working. Run `pip3 install -U pip` once, then upgrade or uninstall.
|
|
472
|
+
- With `--user` the same happens, except that `pip3 install --user -U multiCMD` with no version does not stop: it installs the new release, which works, and leaves the old release's metadata behind. `pip3 list` then shows the old release after `pip3 uninstall multiCMD`, and the next uninstall stops with that error. To avoid this, run `pip3 install --user -U pip` once first. If it has already happened, run that, run `pip3 uninstall multiCMD` until it says multiCMD is not installed, and install again. From 1.58 on the same upgrade leaves the metadata too, but with no error: `pip3 uninstall multiCMD` takes one run per release left.
|
|
473
|
+
- Without `--user`, with multiCMD 1.57 or earlier installed, `pip3 install -U hpcp`, `statblk` or `statbtrfs` stops the same way once a newer multiCMD is out, because pip 9 upgrades dependencies too. `pip3 install -U --no-deps hpcp` gets past it.
|
|
474
|
+
- With pip 10 or later, in a venv whose path has a space or is too long for the kernel's `#!` line, the `multiCMD` spelling does not start (the error varies: "bad interpreter", `import: command not found`), because pip gives a script file a plain `#!` line: use `multicmd`. With pip 9 no spelling starts in such a venv, as in 1.57. uv and system or `--user` installs are not affected.
|
|
475
|
+
- An embedded copy made by the generator now falls back from any failure to import the installed multiCMD, not only the three errors it named.
|
|
476
|
+
|
|
477
|
+
## Upgrading to 1.57 from 1.56
|
|
478
|
+
|
|
479
|
+
- Faster command starts on Python 3.6 - 3.10: the wrapper that puts a command in its own process group
|
|
480
|
+
imports only built-in modules, halving its CPU cost (about 7 ms to 3.5 ms per command on EL8's Python 3.6.8).
|
|
481
|
+
Nothing else about starting a command changes.
|
|
482
|
+
- Fix: `AsyncExecutor`'s `task.thread` is set by the command's thread once it runs. Before, it was set just before
|
|
483
|
+
the thread started, and a `task.thread.join()` in that gap returned at once, with the command not yet run.
|
|
484
|
+
`task.thread` from `run_command(..., wait_for_return=False)` is unchanged.
|
|
485
|
+
|
|
468
486
|
## Upgrading to 1.56 from 1.55
|
|
469
487
|
|
|
470
488
|
- Fix: a command that times out while it redraws a `\r` progress line on stderr keeps its `Timeout!`
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: multiCMD
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.58
|
|
4
4
|
Summary: Run commands simultaneously
|
|
5
5
|
Home-page: https://github.com/yufei-pan/multiCMD
|
|
6
6
|
Author: Yufei Pan
|
|
@@ -56,7 +56,7 @@ It works both as a command-line tool and as an importable wrapper around
|
|
|
56
56
|
pip install multiCMD
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
This installs the module and three equivalent
|
|
59
|
+
This installs the module and three equivalent commands:
|
|
60
60
|
|
|
61
61
|
```bash
|
|
62
62
|
mcmd
|
|
@@ -491,6 +491,24 @@ pytest -m "not slow" # skip intentional sleep-heavy cases
|
|
|
491
491
|
|
|
492
492
|
`test_expand_ranges_perf.py` is a manual performance microbenchmark and is not collected by pytest.
|
|
493
493
|
|
|
494
|
+
## Upgrading to 1.58 from 1.57
|
|
495
|
+
|
|
496
|
+
- The `multiCMD` command is now a script file, not an entry point, so `entry_points.txt` no longer has two names that differ only in case. All three commands still work the same way, except in the venv case below. From 1.58 on, stock pip 9 on EL7 and EL8 upgrades and uninstalls multiCMD without `configparser.DuplicateOptionError`.
|
|
497
|
+
- With pip 9, the pip of EL7 and EL8 (`pip3 --version`), upgrading from 1.57 or earlier and `pip3 uninstall multiCMD` of those releases stop with `configparser.DuplicateOptionError: ... option 'multicmd' in section 'console_scripts' already exists`, and nothing changes: the installed release keeps working. Run `pip3 install -U pip` once, then upgrade or uninstall.
|
|
498
|
+
- With `--user` the same happens, except that `pip3 install --user -U multiCMD` with no version does not stop: it installs the new release, which works, and leaves the old release's metadata behind. `pip3 list` then shows the old release after `pip3 uninstall multiCMD`, and the next uninstall stops with that error. To avoid this, run `pip3 install --user -U pip` once first. If it has already happened, run that, run `pip3 uninstall multiCMD` until it says multiCMD is not installed, and install again. From 1.58 on the same upgrade leaves the metadata too, but with no error: `pip3 uninstall multiCMD` takes one run per release left.
|
|
499
|
+
- Without `--user`, with multiCMD 1.57 or earlier installed, `pip3 install -U hpcp`, `statblk` or `statbtrfs` stops the same way once a newer multiCMD is out, because pip 9 upgrades dependencies too. `pip3 install -U --no-deps hpcp` gets past it.
|
|
500
|
+
- With pip 10 or later, in a venv whose path has a space or is too long for the kernel's `#!` line, the `multiCMD` spelling does not start (the error varies: "bad interpreter", `import: command not found`), because pip gives a script file a plain `#!` line: use `multicmd`. With pip 9 no spelling starts in such a venv, as in 1.57. uv and system or `--user` installs are not affected.
|
|
501
|
+
- An embedded copy made by the generator now falls back from any failure to import the installed multiCMD, not only the three errors it named.
|
|
502
|
+
|
|
503
|
+
## Upgrading to 1.57 from 1.56
|
|
504
|
+
|
|
505
|
+
- Faster command starts on Python 3.6 - 3.10: the wrapper that puts a command in its own process group
|
|
506
|
+
imports only built-in modules, halving its CPU cost (about 7 ms to 3.5 ms per command on EL8's Python 3.6.8).
|
|
507
|
+
Nothing else about starting a command changes.
|
|
508
|
+
- Fix: `AsyncExecutor`'s `task.thread` is set by the command's thread once it runs. Before, it was set just before
|
|
509
|
+
the thread started, and a `task.thread.join()` in that gap returned at once, with the command not yet run.
|
|
510
|
+
`task.thread` from `run_command(..., wait_for_return=False)` is unchanged.
|
|
511
|
+
|
|
494
512
|
## Upgrading to 1.56 from 1.55
|
|
495
513
|
|
|
496
514
|
- Fix: a command that times out while it redraws a `\r` progress line on stderr keeps its `Timeout!`
|
|
@@ -7,6 +7,7 @@ multiCMD.egg-info/SOURCES.txt
|
|
|
7
7
|
multiCMD.egg-info/dependency_links.txt
|
|
8
8
|
multiCMD.egg-info/entry_points.txt
|
|
9
9
|
multiCMD.egg-info/top_level.txt
|
|
10
|
+
scripts/multiCMD
|
|
10
11
|
tests/test_api.py
|
|
11
12
|
tests/test_async_executor.py
|
|
12
13
|
tests/test_cli.py
|
|
@@ -17,6 +18,7 @@ tests/test_env.py
|
|
|
17
18
|
tests/test_expand_ranges.py
|
|
18
19
|
tests/test_helpers.py
|
|
19
20
|
tests/test_interrupts.py
|
|
21
|
+
tests/test_packaging.py
|
|
20
22
|
tests/test_parsing.py
|
|
21
23
|
tests/test_ping.py
|
|
22
24
|
tests/test_pool.py
|
|
@@ -27,9 +27,9 @@ import traceback
|
|
|
27
27
|
from pprint import pformat
|
|
28
28
|
|
|
29
29
|
#%% global vars
|
|
30
|
-
version = '1.
|
|
30
|
+
version = '1.58'
|
|
31
31
|
__version__ = version
|
|
32
|
-
COMMIT_DATE = '2026-
|
|
32
|
+
COMMIT_DATE = '2026-11-06'
|
|
33
33
|
# fire-and-forget threads (run_commands with wait_for_return=False) that join_threads() waits on. multiCMD takes no
|
|
34
34
|
# lock for it: each update is one set operation and each snapshot one set.copy(), as for _live_procs below
|
|
35
35
|
__running_threads = set()
|
|
@@ -52,7 +52,12 @@ _IS_POSIX = os.name == 'posix'
|
|
|
52
52
|
# Popen(process_group=) exists from Python 3.11; older Pythons start commands through _PGROUP_EXEC
|
|
53
53
|
_HAS_PROCESS_GROUP = sys.version_info >= (3, 11)
|
|
54
54
|
# exec wrapper for Python < 3.11: joins a new process group, then becomes the command.
|
|
55
|
-
# preexec_fn
|
|
55
|
+
# Not preexec_fn, tried and measured ( it would save the wrapper's interpreter start ): on 3.6 the forked child frees
|
|
56
|
+
# the other threads' top frames, so the destructors of their locals run in the child and flush the buffers it
|
|
57
|
+
# duplicated, writing a caller's open file twice. On 3.7 - 3.10 preexec_fn runs the at-fork hooks, Python code in the
|
|
58
|
+
# parent: a Ctrl-C at the start of logging's _releaseLock can leave its lock held, and every later command started on
|
|
59
|
+
# another thread hangs on it.
|
|
60
|
+
# Not a new session ( setsid ) either: that would hide the terminal from sudo.
|
|
56
61
|
# It runs in this process's own environment ( so it starts wherever this Python does, LD_LIBRARY_PATH and the
|
|
57
62
|
# like included ). Its first argument is '-' ( the command inherits the wrapper's environment as it is ) or N:
|
|
58
63
|
# the command's own environment is then the N NAME=VALUE entries in the wrapper's variables
|
|
@@ -62,36 +67,42 @@ _HAS_PROCESS_GROUP = sys.version_info >= (3, 11)
|
|
|
62
67
|
# in the wrapper's environment: only a command started with '-' inherits that ( as it always has ).
|
|
63
68
|
_ENV_VAR = '_MULTICMD_ENV_'
|
|
64
69
|
_PGROUP_EXEC = (
|
|
65
|
-
|
|
70
|
+
# only built-in modules: `import os, signal` is half of what the wrapper costs ( signal pulls in enum, os the path
|
|
71
|
+
# modules ), so the os helpers are spelled out for POSIX, the only place the wrapper runs
|
|
72
|
+
"import posix,_signal,sys\n"
|
|
66
73
|
# Python ignores SIGPIPE / SIGXFSZ at startup and an ignored signal survives exec; give the command the defaults
|
|
67
74
|
"for n in ('SIGPIPE','SIGXFSZ'):\n"
|
|
68
|
-
"\tif hasattr(
|
|
69
|
-
"
|
|
75
|
+
"\tif hasattr(_signal,n):_signal.signal(getattr(_signal,n),_signal.SIG_DFL)\n"
|
|
76
|
+
"posix.setpgid(0,0)\n"
|
|
70
77
|
"a=sys.argv\n"
|
|
71
78
|
"c=a[2]\n"
|
|
72
79
|
"v=None\n"
|
|
73
80
|
"if a[1]!='-':\n"
|
|
74
81
|
"\ttry:\n"
|
|
75
|
-
# read, not pop: the command is started with exactly the environment built here
|
|
76
|
-
"\t\tv=dict(
|
|
82
|
+
# read, not pop: the command is started with exactly the environment built here
|
|
83
|
+
"\t\tv=dict(posix.environ[b'_MULTICMD_ENV_%d'%i].split(b'=',1) for i in range(int(a[1])))\n"
|
|
77
84
|
"\texcept(KeyError,ValueError):\n"
|
|
78
85
|
"\t\tsys.stderr.write('multiCMD: incomplete environment for %r\\n'%c);sys.exit(126)\n"
|
|
79
86
|
"def x(f):\n"
|
|
80
|
-
"\tif v is None:
|
|
81
|
-
"\
|
|
87
|
+
"\tif v is None:posix.execv(f,a[2:])\n"
|
|
88
|
+
"\tposix.execve(f,a[2:],v)\n"
|
|
89
|
+
"def g(f):\n"
|
|
90
|
+
"\ttry:return posix.stat(f).st_mode&0o170000==0o100000\n"
|
|
91
|
+
"\texcept(OSError,ValueError):return False\n"
|
|
82
92
|
"try:\n"
|
|
83
|
-
"\tif
|
|
93
|
+
"\tif '/' in c:x(c)\n"
|
|
84
94
|
# a bare name: try the regular files by that name in the PATH directories in turn ( see _spawn ); the first
|
|
85
95
|
# failure other than ENOENT / ENOTDIR says why it cannot run, and without one it is not found
|
|
86
|
-
"\tp=
|
|
96
|
+
"\tp=(posix.environ if v is None else v).get(b'PATH')\n"
|
|
87
97
|
"\tr=None\n"
|
|
88
|
-
"\tfor d in
|
|
89
|
-
"\t\
|
|
90
|
-
"\t\
|
|
98
|
+
"\tfor d in ("+repr(os.defpath)+" if p is None else p.decode(sys.getfilesystemencoding(),'surrogateescape')).split(':'):\n"
|
|
99
|
+
"\t\td=d or '.'\n"
|
|
100
|
+
"\t\tf=d+c if d.endswith('/') else d+'/'+c\n"
|
|
101
|
+
"\t\tif g(f):\n"
|
|
91
102
|
"\t\t\ttry:x(f)\n"
|
|
92
103
|
"\t\t\texcept(FileNotFoundError,NotADirectoryError):pass\n"
|
|
93
104
|
"\t\t\texcept OSError as e:r=r or e\n"
|
|
94
|
-
"\traise r or FileNotFoundError(2,
|
|
105
|
+
"\traise r or FileNotFoundError(2,posix.strerror(2),c)\n"
|
|
95
106
|
# a plain command reports as it always has; one with an environment as Popen does ( Python 3.11+: the error with the command as given )
|
|
96
107
|
"except OSError as e:\n"
|
|
97
108
|
"\tsys.stderr.write((str(e) if a[1]=='-' else str(OSError(e.errno,e.strerror,c)))+'\\n')\n"
|
|
@@ -505,21 +516,16 @@ class AsyncExecutor:
|
|
|
505
516
|
def _track(self, thread):
|
|
506
517
|
'''
|
|
507
518
|
Track a _Thread about to start until it reports itself finished ( internal )
|
|
508
|
-
Tracked before it starts, so it is tracked before it can report itself finished
|
|
509
|
-
|
|
519
|
+
Tracked before it starts, so it is tracked before it can report itself finished. Its task gets it from
|
|
520
|
+
_run_executor_task: a task.thread still starting would look finished, as its join() returns at once.
|
|
510
521
|
'''
|
|
511
522
|
self._threads[thread] = None
|
|
512
|
-
if thread.task is not None:
|
|
513
|
-
thread.task.thread = thread
|
|
514
523
|
|
|
515
524
|
def _untrack(self, thread):
|
|
516
525
|
'''
|
|
517
526
|
Stop tracking a thread that never started, as it never reports itself finished ( internal )
|
|
518
|
-
Its task drops it as a finished one does: an executor task has a thread only while it runs.
|
|
519
527
|
'''
|
|
520
528
|
self._threads.pop(thread, None)
|
|
521
|
-
if thread.task is not None and thread.task.thread is thread:
|
|
522
|
-
thread.task.thread = None
|
|
523
529
|
|
|
524
530
|
def _dispatch(self):
|
|
525
531
|
'''
|
|
@@ -2066,7 +2072,9 @@ def _fail_task(task, message, error, quiet=False):
|
|
|
2066
2072
|
def _run_executor_task(task, permit, timeout, quiet, dry_run, finished, decor=('', '', ''), use_pty=False):
|
|
2067
2073
|
'''
|
|
2068
2074
|
Run an AsyncExecutor command, then report this thread finished so the executor joins it and drops it from the task ( internal )
|
|
2075
|
+
An executor task has a thread only while it runs, so task.thread is set here, by the running thread.
|
|
2069
2076
|
'''
|
|
2077
|
+
task.thread = threading.current_thread()
|
|
2070
2078
|
try:
|
|
2071
2079
|
__run_command(task, permit, timeout, quiet, dry_run, decor=decor, use_pty=use_pty)
|
|
2072
2080
|
finally:
|
|
@@ -11,10 +11,12 @@ setup(
|
|
|
11
11
|
author_email='pan@zopyr.us',
|
|
12
12
|
url='https://github.com/yufei-pan/multiCMD',
|
|
13
13
|
py_modules=['multiCMD'],
|
|
14
|
+
# The mixed-case command is a script file and not an entry point: pip 9 (stock on EL7 and EL8) reads entry_points.txt with a
|
|
15
|
+
# parser that folds case, so two commands that differ only in case make it fail to upgrade or uninstall the package.
|
|
16
|
+
scripts=['scripts/multiCMD'],
|
|
14
17
|
entry_points={
|
|
15
18
|
'console_scripts': [
|
|
16
19
|
'mcmd=multiCMD:main',
|
|
17
|
-
'multiCMD=multiCMD:main',
|
|
18
20
|
'multicmd=multiCMD:main',
|
|
19
21
|
],
|
|
20
22
|
},
|
|
@@ -352,6 +352,30 @@ def _started_thread(task):
|
|
|
352
352
|
return task.thread
|
|
353
353
|
|
|
354
354
|
|
|
355
|
+
def test_a_tasks_thread_shows_only_once_it_runs(monkeypatch):
|
|
356
|
+
# task.thread is None until the command's thread runs: a thread still starting joins at once, before the command ran
|
|
357
|
+
real = multiCMD._Thread.start
|
|
358
|
+
reached, go = threading.Event(), threading.Event()
|
|
359
|
+
def held(self):
|
|
360
|
+
if self.task is not None:
|
|
361
|
+
reached.set()
|
|
362
|
+
go.wait(5)
|
|
363
|
+
real(self)
|
|
364
|
+
monkeypatch.setattr(multiCMD._Thread, 'start', held)
|
|
365
|
+
ex = AsyncExecutor(max_threads=1, quiet=True)
|
|
366
|
+
try:
|
|
367
|
+
task = ex.run_command(['sleep', '0.3'])
|
|
368
|
+
assert reached.wait(5)
|
|
369
|
+
assert task.thread is None
|
|
370
|
+
go.set()
|
|
371
|
+
thread = _started_thread(task)
|
|
372
|
+
thread.join()
|
|
373
|
+
assert task.returncode == 0
|
|
374
|
+
finally:
|
|
375
|
+
go.set()
|
|
376
|
+
ex.cleanup()
|
|
377
|
+
|
|
378
|
+
|
|
355
379
|
def test_sigint_helper_fails_a_call_that_returns_first_and_sends_nothing_later():
|
|
356
380
|
start = time.monotonic()
|
|
357
381
|
with pytest.raises(pytest.fail.Exception, match='returned before the SIGINT'):
|
|
@@ -708,13 +708,10 @@ def test_an_internal_oserror_is_still_an_error_with_a_traceback(launcher, monkey
|
|
|
708
708
|
proc.stderr.close()
|
|
709
709
|
|
|
710
710
|
|
|
711
|
-
def
|
|
712
|
-
|
|
713
|
-
assert 'SIGXFZ' not in multiCMD._PGROUP_EXEC and '.pop(' not in multiCMD._PGROUP_EXEC
|
|
714
|
-
# ( Python before 3.9 calls os.environb.unsetenv, 3.9+ the module's os.unsetenv )
|
|
715
|
-
code = ("import os, sys\nos.environb.unsetenv = os.unsetenv = lambda *a: sys.exit(99)\n"
|
|
711
|
+
def test_the_wrapper_starts_the_command_with_exactly_the_environment_it_was_given():
|
|
712
|
+
code = ("import posix, sys\n"
|
|
716
713
|
"def execve(f, a, e):\n\tprint(sorted(e.items()))\n\tsys.exit(0)\n"
|
|
717
|
-
"
|
|
714
|
+
"posix.execve = execve\nexec(%r)" % multiCMD._PGROUP_EXEC)
|
|
718
715
|
env = {'PATH': os.environ['PATH'], '_MULTICMD_ENV_0': 'A=1', '_MULTICMD_ENV_1': 'B=x=y'}
|
|
719
716
|
proc = subprocess.run([sys.executable, '-I', '-S', '-c', code, '2', 'true'], env=env,
|
|
720
717
|
stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=30)
|
|
@@ -722,6 +719,18 @@ def test_the_wrapper_does_not_unset_each_variable_it_reads():
|
|
|
722
719
|
assert proc.stdout.strip() == "[(b'A', b'1'), (b'B', b'x=y')]"
|
|
723
720
|
|
|
724
721
|
|
|
722
|
+
@pytest.mark.parametrize('count', ['-', '0'], ids=['plain', 'environment'])
|
|
723
|
+
def test_the_wrapper_starts_nothing_but_built_in_modules(count):
|
|
724
|
+
# every command on Python < 3.11 pays for what the wrapper imports: os and signal ( with enum ) double its cost
|
|
725
|
+
code = ("import posix, sys\nbefore = set(sys.modules)\n"
|
|
726
|
+
"def run(*a):\n\tprint(sorted(set(sys.modules) - before))\n\tsys.exit(0)\n"
|
|
727
|
+
"posix.execv = posix.execve = run\nexec(%r)" % multiCMD._PGROUP_EXEC)
|
|
728
|
+
proc = subprocess.run([sys.executable, '-I', '-S', '-c', code, count, 'sh'], env={'PATH': '/usr/bin:/bin'},
|
|
729
|
+
stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=30)
|
|
730
|
+
assert proc.returncode == 0, proc.stderr
|
|
731
|
+
assert proc.stdout.strip() == '[]'
|
|
732
|
+
|
|
733
|
+
|
|
725
734
|
def test_tasks_that_never_started_do_not_keep_their_environment():
|
|
726
735
|
ex = AsyncExecutor(max_threads=1, quiet=True)
|
|
727
736
|
blocker = ex.run_command(['sh', '-c', 'sleep 0.5'])
|
|
@@ -813,6 +822,11 @@ def _sh_status(path, command):
|
|
|
813
822
|
return subprocess.run(['/bin/sh', '-c', command], env={'PATH': path}, stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode
|
|
814
823
|
|
|
815
824
|
|
|
825
|
+
# multiCMD does as dash: a fifo in PATH, or a script whose interpreter is missing, is not found ( 127 ). bash as /bin/sh
|
|
826
|
+
# finds the file and says it cannot be run ( 126, and 127 for a missing interpreter from bash 5.2 ): the tests only check that sh does
|
|
827
|
+
# not run it either, so the exact value of sh's own code is not pinned
|
|
828
|
+
|
|
829
|
+
|
|
816
830
|
@pytest.mark.parametrize('kind', ['directory', 'fifo', 'unsearchable'])
|
|
817
831
|
@pytest.mark.parametrize('how', ['plain', 'extra_env', 'env'])
|
|
818
832
|
def test_a_path_entry_without_a_regular_file_by_that_name_is_not_found(launcher, tmp_path, monkeypatch, kind, how):
|
|
@@ -833,7 +847,7 @@ def test_a_path_entry_without_a_regular_file_by_that_name_is_not_found(launcher,
|
|
|
833
847
|
task = _run_in_path(monkeypatch, how, path, command)
|
|
834
848
|
assert task.returncode == 127 and not task._error, task.stderr
|
|
835
849
|
assert "No such file or directory: '%s'" % command in ' '.join(task.stderr)
|
|
836
|
-
assert _sh_status(path, command) == 127
|
|
850
|
+
assert _sh_status(path, command) in ((126, 127) if kind == 'fifo' else (127,))
|
|
837
851
|
finally:
|
|
838
852
|
where.chmod(0o755)
|
|
839
853
|
|
|
@@ -851,7 +865,7 @@ def test_a_script_whose_interpreter_is_missing_is_not_found_whatever_path_follow
|
|
|
851
865
|
task = _run_in_path(monkeypatch, how, path, 'badinterp')
|
|
852
866
|
assert task.returncode == 127 and not task._error, task.stderr
|
|
853
867
|
assert "No such file or directory: 'badinterp'" in ' '.join(task.stderr)
|
|
854
|
-
assert _sh_status(path, 'badinterp')
|
|
868
|
+
assert _sh_status(path, 'badinterp') in (126, 127)
|
|
855
869
|
|
|
856
870
|
|
|
857
871
|
@pytest.mark.parametrize('how', ['plain', 'extra_env', 'env'])
|
|
@@ -129,7 +129,7 @@ def _left_running(before, wait=6.0):
|
|
|
129
129
|
time.sleep(0.02)
|
|
130
130
|
|
|
131
131
|
|
|
132
|
-
# on Python 3.6 - 3.9 and 3.14 a Ctrl-C
|
|
132
|
+
# on Python 3.6 - 3.9 and 3.14 a Ctrl-C at one instruction of a `with lock:` can leave the lock held, and
|
|
133
133
|
# Thread.start() takes threading's own _active_limbo_lock so: every thread then hangs as it ends ( CPython's own, see
|
|
134
134
|
# the README ). The stress test below keeps its Ctrl-Cs out of the output readers' Thread.start() there: one that would
|
|
135
135
|
# land inside it is taken right after it
|
|
@@ -553,8 +553,8 @@ def test_a_ctrl_c_at_any_point_of_a_serial_run_is_raised(command):
|
|
|
553
553
|
# ( matrix of round 12, finding 4 ) one KeyboardInterrupt at each call, return and C-call return of a serial run in
|
|
554
554
|
# turn, in every frame of the main thread ( subprocess's, os's ... ): run_command() raises it every time, for a
|
|
555
555
|
# command that runs and for one that is not found. It found a Ctrl-C that Thread.start() turns into RuntimeError on
|
|
556
|
-
# 3.12 ( see _Thread.start ). threading's own frames are left out where a Ctrl-C
|
|
557
|
-
#
|
|
556
|
+
# 3.12 ( see _Thread.start ). threading's own frames are left out where a Ctrl-C at one instruction of a `with lock:`
|
|
557
|
+
# can leave the lock held ( 3.6 - 3.9, 3.14: CPython's own, see the README )
|
|
558
558
|
skipped = {threading.__file__} if _WITH_MAY_JAM else set()
|
|
559
559
|
def counted(frame, event):
|
|
560
560
|
return frame.f_code.co_filename not in skipped
|
|
@@ -575,7 +575,8 @@ def test_a_ctrl_c_at_any_point_of_a_serial_run_is_raised(command):
|
|
|
575
575
|
break
|
|
576
576
|
state['k'] += 1
|
|
577
577
|
assert returned == []
|
|
578
|
-
|
|
578
|
+
# ( the matrix went through the run: 'missing' on 3.6 has the fewest points, some 70 and 10 per PATH entry looked in )
|
|
579
|
+
assert state['k'] > 50 and not multiCMD._live_procs
|
|
579
580
|
|
|
580
581
|
|
|
581
582
|
@pytest.mark.skipif(not sys.platform.startswith('linux'), reason='POSIX commands')
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
"""The packaging: the commands setup.py installs."""
|
|
2
|
+
import os
|
|
3
|
+
import subprocess
|
|
4
|
+
import sys
|
|
5
|
+
from pathlib import Path
|
|
6
|
+
|
|
7
|
+
import pytest
|
|
8
|
+
|
|
9
|
+
PACKAGE_ROOT = Path(__file__).resolve().parents[1]
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
def test_no_two_commands_differ_only_in_case(tmp_path):
|
|
13
|
+
pytest.importorskip("setuptools", reason="needs setuptools to run setup.py")
|
|
14
|
+
# pip 9 (stock on EL7 and EL8) reads entry_points.txt with a parser that folds case, so two commands that differ only
|
|
15
|
+
# in case make it fail to upgrade or uninstall the package. The mixed-case command is a script file (scripts=) instead.
|
|
16
|
+
result = subprocess.run([sys.executable, "setup.py", "-q", "egg_info", "--egg-base", str(tmp_path)],
|
|
17
|
+
cwd=str(PACKAGE_ROOT), env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"), stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
|
18
|
+
universal_newlines=True, timeout=120)
|
|
19
|
+
assert result.returncode == 0, result.stderr
|
|
20
|
+
egg_info = tmp_path / "multiCMD.egg-info"
|
|
21
|
+
entry_points = (egg_info / "entry_points.txt").read_text(encoding="utf-8")
|
|
22
|
+
names = [line.split("=")[0].strip() for line in entry_points.splitlines() if "=" in line]
|
|
23
|
+
assert sorted(names) == ["mcmd", "multicmd"]
|
|
24
|
+
# the mixed-case command ships in the sdist as a script, so the three spellings stay three commands
|
|
25
|
+
assert "scripts/multiCMD" in (egg_info / "SOURCES.txt").read_text(encoding="utf-8").splitlines()
|
|
26
|
+
assert (PACKAGE_ROOT / "scripts" / "multiCMD").read_text(encoding="utf-8").startswith("#!")
|
|
@@ -49,7 +49,8 @@ TTY_PROBE = textwrap.dedent(r"""
|
|
|
49
49
|
sys.path.insert(0, {root!r})
|
|
50
50
|
import multiCMD
|
|
51
51
|
multiCMD._HAS_PROCESS_GROUP = {native!r}
|
|
52
|
-
|
|
52
|
+
# ( 2> comes second: bash reports the failed read, which is the point here, but a failed open still shows )
|
|
53
|
+
proc, pgid = multiCMD._spawn(['sh', '-c', 'read x </dev/tty 2>/dev/null; echo rc=$?'], subprocess.PIPE, subprocess.STDOUT)
|
|
53
54
|
try:
|
|
54
55
|
print('RESULT', proc.communicate(timeout=5)[0].decode().strip())
|
|
55
56
|
except subprocess.TimeoutExpired:
|
|
@@ -66,10 +66,20 @@ def test_executor_stop_kills_grandchildren_promptly():
|
|
|
66
66
|
assert pid_gone(int(task.stdout[0]))
|
|
67
67
|
|
|
68
68
|
|
|
69
|
-
def test_timing_fields():
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
69
|
+
def test_timing_fields(tmp_path):
|
|
70
|
+
# no timing margin: the command runs until the test says so, and the test says so only once start_time is set
|
|
71
|
+
go = tmp_path / 'go'
|
|
72
|
+
t0 = time.time()
|
|
73
|
+
task = run_commands([['sh', '-c', 'until [ -e "$0" ]; do sleep 0.01; done', str(go)]], quiet=True, return_object=True, wait_for_return=False)[0]
|
|
74
|
+
deadline = time.monotonic() + 30
|
|
75
|
+
while task.start_time is None and time.monotonic() < deadline:
|
|
76
|
+
time.sleep(0.01)
|
|
77
|
+
during = time.time()
|
|
78
|
+
go.touch()
|
|
79
|
+
multiCMD.join_threads()
|
|
80
|
+
t1 = time.time()
|
|
81
|
+
assert t0 <= task.start_time <= during <= task.end_time <= t1
|
|
82
|
+
assert task.duration == task.end_time - task.start_time
|
|
73
83
|
assert multiCMD.Task(['x']).duration is None
|
|
74
84
|
|
|
75
85
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|