multiCMD 1.54__tar.gz → 1.55__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. {multicmd-1.54 → multicmd-1.55}/PKG-INFO +138 -11
  2. {multicmd-1.54 → multicmd-1.55}/README.md +137 -10
  3. {multicmd-1.54 → multicmd-1.55}/multiCMD.egg-info/PKG-INFO +138 -11
  4. {multicmd-1.54 → multicmd-1.55}/multiCMD.egg-info/SOURCES.txt +2 -0
  5. multicmd-1.55/multiCMD.py +3849 -0
  6. multicmd-1.55/tests/test_async_executor.py +1071 -0
  7. multicmd-1.55/tests/test_env.py +965 -0
  8. multicmd-1.55/tests/test_pool.py +218 -0
  9. multicmd-1.55/tests/test_process_groups.py +394 -0
  10. {multicmd-1.54 → multicmd-1.55}/tests/test_pty.py +28 -0
  11. multicmd-1.55/tests/test_settle.py +2412 -0
  12. multicmd-1.55/tests/test_supervision.py +500 -0
  13. multicmd-1.54/multiCMD.py +0 -2354
  14. multicmd-1.54/tests/test_async_executor.py +0 -251
  15. multicmd-1.54/tests/test_pool.py +0 -70
  16. multicmd-1.54/tests/test_process_groups.py +0 -155
  17. multicmd-1.54/tests/test_supervision.py +0 -207
  18. {multicmd-1.54 → multicmd-1.55}/LICENSE +0 -0
  19. {multicmd-1.54 → multicmd-1.55}/multiCMD.egg-info/dependency_links.txt +0 -0
  20. {multicmd-1.54 → multicmd-1.55}/multiCMD.egg-info/entry_points.txt +0 -0
  21. {multicmd-1.54 → multicmd-1.55}/multiCMD.egg-info/top_level.txt +0 -0
  22. {multicmd-1.54 → multicmd-1.55}/setup.cfg +0 -0
  23. {multicmd-1.54 → multicmd-1.55}/setup.py +0 -0
  24. {multicmd-1.54 → multicmd-1.55}/tests/test_api.py +0 -0
  25. {multicmd-1.54 → multicmd-1.55}/tests/test_cli.py +0 -0
  26. {multicmd-1.54 → multicmd-1.55}/tests/test_cli_input.py +0 -0
  27. {multicmd-1.54 → multicmd-1.55}/tests/test_cli_summary.py +0 -0
  28. {multicmd-1.54 → multicmd-1.55}/tests/test_display.py +0 -0
  29. {multicmd-1.54 → multicmd-1.55}/tests/test_expand_ranges.py +0 -0
  30. {multicmd-1.54 → multicmd-1.55}/tests/test_helpers.py +0 -0
  31. {multicmd-1.54 → multicmd-1.55}/tests/test_parsing.py +0 -0
  32. {multicmd-1.54 → multicmd-1.55}/tests/test_ping.py +0 -0
  33. {multicmd-1.54 → multicmd-1.55}/tests/test_reader.py +0 -0
  34. {multicmd-1.54 → multicmd-1.55}/tests/test_regressions.py +0 -0
  35. {multicmd-1.54 → multicmd-1.55}/tests/test_run_command.py +0 -0
  36. {multicmd-1.54 → multicmd-1.55}/tests/test_run_commands.py +0 -0
  37. {multicmd-1.54 → multicmd-1.55}/tests/test_sudo.py +0 -0
  38. {multicmd-1.54 → multicmd-1.55}/tests/test_task.py +0 -0
  39. {multicmd-1.54 → multicmd-1.55}/tests/test_windows.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: multiCMD
3
- Version: 1.54
3
+ Version: 1.55
4
4
  Summary: Run commands simultaneously
5
5
  Home-page: https://github.com/yufei-pan/multiCMD
6
6
  Author: Yufei Pan
@@ -71,8 +71,8 @@ You can also just drop `multiCMD.py` into your project and import it directly.
71
71
  ```bash
72
72
  $ mcmd -h
73
73
  usage: mcmd [-h] [-p] [-t SECONDS] [-m max_threads] [--sudo] [-q] [-n]
74
- [--shell] [-f FILE] [-a FILE] [--summary | --no-summary]
75
- [--no-prefix] [--no-color] [--pty | --no-pty] [-V]
74
+ [--shell] [-e NAME=VALUE] [-f FILE] [-a FILE] [--summary |
75
+ --no-summary] [--no-prefix] [--no-color] [--pty | --no-pty] [-V]
76
76
  [command ...]
77
77
 
78
78
  Run multiple commands in parallel
@@ -96,6 +96,8 @@ options:
96
96
  -n, --dry-run print each command, quoted, instead of running it
97
97
  --shell run each command through /bin/sh -c ( pipes, ;, > and
98
98
  globs work )
99
+ -e, --env NAME=VALUE set NAME to VALUE in the environment of every command
100
+ ( repeatable ); NAME alone removes it
99
101
  -f, --file FILE read commands from FILE, one per line ( - for stdin;
100
102
  blank lines and # comments are skipped )
101
103
  -a, --arg-file FILE run each command template once per line of FILE, with
@@ -293,10 +295,47 @@ multiCMD.run_commands(cmds, max_threads=8, color=False) # never color
293
295
  multiCMD.run_command(["python3", "train.py"], pty=True) # live output, stderr merged
294
296
  ```
295
297
 
298
+ ### Environment per call
299
+
300
+ ```python
301
+ multiCMD.run_command(["xfs_info", "/data"], extra_env={"LC_ALL": "C"}) # add / override variables
302
+ multiCMD.run_command(["env"], extra_env={"HTTP_PROXY": None}) # None removes one
303
+ multiCMD.run_command(["env"], env={"PATH": "/usr/bin", "HOME": "/tmp"}) # the whole environment, as subprocess's env
304
+ ```
305
+
306
+ `env` is the whole environment of the command, as `subprocess`'s `env` ( names and values
307
+ are `str`; on POSIX `bytes` and `os.PathLike` work too ). `extra_env` is applied on top of the
308
+ inherited environment, or of `env` when both are given; a `None` value removes the variable,
309
+ and an empty `extra_env` is the same as none. "Inherited" is `os.environ` as the process holds
310
+ it when the command starts: a variable set behind Python's back ( `os.putenv`, `os.unsetenv`, a
311
+ C library's `setenv` ) reaches commands started without `env` / `extra_env`, not the ones
312
+ given them. An inherited variable with an empty name ( a corrupt `environ` entry such as `=x` ) is left out of the environment of a command given `env` / `extra_env`, and out of an `env` copied from `os.environ`; an empty name in `extra_env` is a `ValueError`. Names are case-insensitive on Windows, so `extra_env={"Path": ...}` replaces `PATH`
313
+ there. On POSIX a command is looked up in the `PATH` of the environment it gets, on every Python
314
+ version; Windows searches the calling process's `PATH` whatever the command's environment says.
315
+ Before Python 3.11 the launcher starts in this process's environment and is handed the command's on
316
+ top of it, so the two together must fit the system's argument limit ( `getconf ARG_MAX`; 2 MiB on Linux
317
+ with the default stack, 128 KiB for one string: before Python 3.11 an entry travels as `_MULTICMD_ENV_<i>=NAME=VALUE`, 15 bytes
318
+ plus the digits of `i` more than its `NAME=VALUE` ); one that does not fit fails with return
319
+ code 126, `Argument list too long`, on every Python.
320
+ On Python 3.7-3.10 a command started with
321
+ neither `env` nor `extra_env` still gets the `LC_CTYPE=C.UTF-8` that interpreter's locale coercion
322
+ sets in a C locale ( as before ); a command given one of them gets exactly the environment described.
323
+ Both are per command: they never change `os.environ` and never reach another command, so
324
+ concurrent commands each see their own. `AsyncExecutor` takes them too, as defaults a call
325
+ overrides. `sudo` resets the environment by default, so with `use_sudo` they may not reach the
326
+ command. On the command line: `mcmd -e LC_ALL=C -e UNSET_ME "cmd"`; `-n` shows them as
327
+ `env LC_ALL=C cmd` ( `env -i ...` for a whole environment ).
328
+
296
329
  `raise_error=True` raises `multiCMD.CommandError`; its `.tasks` holds the failed `Task`
297
330
  objects and its message lists each one with its return code and last error line.
298
331
  `Task` also records `start_time`, `end_time` and `duration`. With `dry_run=True` nothing
299
332
  runs; each command is printed once, quoted so it can be pasted into a shell.
333
+ A command that is not found (return code 127), cannot be executed (126) or that multiCMD
334
+ itself could not run (-1, `task._error`) says why in its `task.stderr`; multiCMD also
335
+ prints a line saying so on stderr, unless `quiet=True`. Before Python 3.11 the exec of a
336
+ command given by its path, or of a file found in `PATH` that may not be executed, happens in
337
+ the launcher's process: it exits 127 / 126 with the reason as the command's own stderr, which
338
+ then shows with the command's output ( unless quiet ) instead of as a line of multiCMD's own.
300
339
 
301
340
  ### Asynchronous / fire-and-forget
302
341
 
@@ -309,7 +348,8 @@ tasks = multiCMD.run_commands(
309
348
  max_threads=2, quiet=True,
310
349
  wait_for_return=False, return_object=True,
311
350
  )
312
- # tasks[i].returncode is None until that command completes
351
+ # tasks[i].returncode is None until that command completes; tasks[i].thread.join(timeout)
352
+ # waits for one command ( a command whose thread could not start fails with -1, its thread already done )
313
353
 
314
354
  # Later, block until everything launched this way has finished:
315
355
  multiCMD.join_threads()
@@ -336,7 +376,8 @@ executor holds at most one thread per permit plus the dispatcher, however many
336
376
  commands are queued. A command's `task.thread` is `None` until it starts, and goes
337
377
  back to `None` once it has finished and the executor has joined its thread, so a
338
378
  large batch does not keep a `Thread` object per finished command. `stop()` skips
339
- commands that have not started yet.
379
+ commands that have not started yet, and so does a Ctrl-C ( or an error ) that interrupts
380
+ `run_command(s)`, for that call's commands.
340
381
 
341
382
  ### Range expansion from Python
342
383
 
@@ -380,11 +421,57 @@ every command inherits through any shell.
380
421
 
381
422
  Because the commands are not in your program's process group, signals sent to that
382
423
  group do not reach them. multiCMD handles this for you: a call waiting on the main
383
- thread that gets Ctrl-C stops its commands before re-raising, `mcmd` stops them on
424
+ thread that gets Ctrl-C, or the SystemExit of a SIGTERM handler's `sys.exit()`
425
+ (`join_threads()` and `AsyncExecutor.wait()` included), stops its commands, the queued ones
426
+ skipped, before re-raising, giving them up to 5 s to exit (a second Ctrl-C stops waiting
427
+ for them at once, a serial run's command still sent SIGTERM; one that lands while multiCMD stops them or forgets the threads that never ran ends
428
+ it too), `mcmd` stops them on
384
429
  SIGTERM / SIGHUP (unless it was started with them ignored, as under `nohup`), and any
385
- commands still running when the interpreter exits are killed. A program that handles
430
+ commands still running when the interpreter exits are killed. From then on a command queued
431
+ before the exit on one of multiCMD's threads does not start: it fails (return code -1, "not started:
432
+ the interpreter is exiting"), as the thread would be frozen with the interpreter and the command
433
+ outlive it; a command run from an `atexit` handler, a `weakref.finalize` callback or a `__del__`
434
+ after that (on the main thread) runs as usual. A program that handles
386
435
  SIGTERM itself, or that runs multiCMD from its own threads (where Ctrl-C never
387
- arrives), can call `multiCMD.kill_all()` from its handler.
436
+ arrives), can call `multiCMD.kill_all()` from its handler. A command still running a second
437
+ after SIGKILL (in uninterruptible sleep, or not yours to signal) is no longer tracked, so later
438
+ calls and the exit do not wait for it again; multiCMD names its pid on stderr, unless quiet.
439
+
440
+ A Ctrl-C (or a signal handler's exception, such as a SIGTERM handler's `sys.exit()`) that lands
441
+ while a command starts stops it like any other, from the moment `Popen` knows the child's pid.
442
+ One that lands while the command is being forked can leave it running unregistered, as in 1.54:
443
+ if Python raises it as the fork returns, before `Popen` has stored the pid, neither the call nor
444
+ `kill_all()` nor the exit hook knows the command, which runs on in its own process group. Python
445
+ raises those on the main thread only, so this concerns serial runs (`max_threads=1`); and the window
446
+ is the fork's own length, some 60-150 µs per command however long it then runs. That is the only place
447
+ one Ctrl-C can lose a command. Before Python 3.11 a second one can too: on Python 3.6-3.10 a second
448
+ Ctrl-C that lands while the first is still on its way out of the command's start, before multiCMD has
449
+ begun to deal with it, is raised before multiCMD has registered the command (one that lands so while a
450
+ call waits for a running command leaves it registered but not signalled: `kill_all()` and the exit hook
451
+ still reach it), and on 3.6-3.9, where Python raises a Ctrl-C at nearly any instruction, a second one at a
452
+ few other moments of that handling can do the same, and even a first one, at a few instructions between the
453
+ steps of a run, can leave a command registered but not signalled. From Python 3.11 no other window is known where
454
+ a Ctrl-C loses a command. On every Python, two Ctrl-Cs at nearly the same moment, the first cutting short the stop
455
+ of a command that timed out or of what a finished command left running with its output open, can leave that
456
+ not signalled: `kill_all()` still reaches it, and so does the exit hook, except on Python 3.6-3.10 for what a
457
+ finished command left running, which can then outlive the program.
458
+ On Python 3.6-3.9 and 3.14 a Ctrl-C that lands just as Python's `threading` takes one of its own
459
+ locks with `with` (as `Thread.start()` does, for a serial run's output readers too) can leave that
460
+ lock held, and the program then hangs as its threads end. That is CPython's own, and any threaded
461
+ program can meet it; Python 3.10-3.13 do not leave the lock held.
462
+
463
+ multiCMD never touches SIGINT's handler (`mcmd` sets its own SIGTERM / SIGHUP handlers, nothing
464
+ else). A Ctrl-C that lands while multiCMD does its bookkeeping on the rare paths (a thread that
465
+ cannot start, stopping commands after a Ctrl-C or a failure, cancelling a batch, recording a command
466
+ that cannot be run) makes that bookkeeping run again from the start, and the KeyboardInterrupt (or
467
+ whatever your own handler raised) is raised once it is complete, so it neither cuts that short nor gets
468
+ lost. Your handler, if you installed one, is called with every Ctrl-C as it lands, and C-level hooks on
469
+ SIGINT (`faulthandler.register`, `signal.siginterrupt`) stay as they are. Whatever your handler raises
470
+ while a call waits for its commands is handled as a Ctrl-C: the commands are stopped (a serial run's
471
+ current one recorded as stopped, the ones after it skipped) and the exception is raised on. multiCMD
472
+ tells it from a failure of its own by your handler's frame in its traceback, and takes anything raised
473
+ while it only waits for a running command as yours; a handler that uninstalls itself before it
474
+ raises is not seen elsewhere, and what it raises then while a command starts counts as a failure.
388
475
 
389
476
  ## Key functions and objects
390
477
 
@@ -392,12 +479,13 @@ arrives), can call `multiCMD.kill_all()` from its handler.
392
479
  run_command(command, timeout=0, max_threads=1, quiet=False, dry_run=False,
393
480
  with_stdErr=False, return_code_only=False, return_object=False,
394
481
  wait_for_return=True, sem=None, use_sudo=..., raise_error=False,
395
- prefix=False, color=None, pty=False, shell=False)
482
+ prefix=False, color=None, pty=False, shell=False, env=None, extra_env=None)
396
483
 
397
484
  run_commands(commands, timeout=0, max_threads=1, quiet=False, dry_run=False,
398
485
  with_stdErr=False, return_code_only=False, return_object=False,
399
486
  parse=False, wait_for_return=True, sem=None, use_sudo=...,
400
- raise_error=False, prefix=False, color=None, pty=False, shell=False)
487
+ raise_error=False, prefix=False, color=None, pty=False, shell=False,
488
+ env=None, extra_env=None)
401
489
 
402
490
  ping(hosts, timeout=1, max_threads=0, ...) # returns True/False reachability
403
491
  join_threads(threads=..., timeout=None) # join fire-and-forget threads
@@ -406,7 +494,8 @@ set_sudo(use_sudo) # enable/disable sudo globally
406
494
 
407
495
  class Task: # command, returncode, stdout, stderr, start_time, end_time, duration
408
496
  class CommandError: # raised by raise_error=True; .tasks = the failed Tasks
409
- class AsyncExecutor: # run_command(s), wait, join, stop, cleanup, get_results, get_return_codes
497
+ class AsyncExecutor: # run_command(s), wait, join, stop, cleanup, get_results, get_return_codes;
498
+ # takes the options above, env / extra_env as defaults a call overrides
410
499
  ```
411
500
 
412
501
  The module also bundles a few terminal/formatting helpers used internally and
@@ -432,6 +521,44 @@ pytest -m "not slow" # skip intentional sleep-heavy cases
432
521
 
433
522
  `test_expand_ranges_perf.py` is a manual performance microbenchmark and is not collected by pytest.
434
523
 
524
+ ## Upgrading to 1.55 from 1.54
525
+
526
+ - New: a per-call environment. `env=` (the whole environment, as `subprocess`'s) and `extra_env=`
527
+ (added on top; a `None` value removes a variable) on `run_command`, `run_commands` and
528
+ `AsyncExecutor` (constructor defaults, overridden per call), and `mcmd -e NAME=VALUE` / `-e NAME`;
529
+ see [Environment per call](#environment-per-call).
530
+ - `quiet=True` now hides the lines about a command that could not be run: `Command path not found`
531
+ (127), `Command cannot be executed` (126) and the error of a command multiCMD could not run (-1),
532
+ which 1.54 printed even when quiet, partly on stdout. The reasons are still in `task.stderr`. Warnings
533
+ about the run's setup (sudo not found or not usable, no pty) still print.
534
+ - A Ctrl-C in `join_threads()` stops the commands it waits for, skips the ones still queued, waits up
535
+ to 5 s for them and re-raises; 1.54 re-raised and left them running. A Ctrl-C that interrupts
536
+ `AsyncExecutor.run_command(s)` skips that call's commands.
537
+ - A SIGTERM handler's `sys.exit()`, or whatever your own signal handler raises while a call waits for
538
+ its commands, stops them and records them as stopped before it is raised on, as a Ctrl-C does; 1.54
539
+ did that only for `KeyboardInterrupt`.
540
+ - A Ctrl-C while a command starts, times out or is stopped, or while multiCMD reads the output that
541
+ a finished command's background processes still hold open, no longer leaves the command or those
542
+ processes running unknown to `kill_all()` and the exit hook, or known to them but never signalled.
543
+ A second Ctrl-C ends the wait for the commands at once; each is still sent SIGTERM, and `kill_all()`
544
+ and the exit hook finish the kill. The limits below are the exceptions.
545
+ - Once the interpreter exits, a command queued on one of multiCMD's threads that has not started fails
546
+ with -1 ("not started: the interpreter is exiting") instead of starting and outliving the program.
547
+ - A command still running a second after SIGKILL is no longer tracked, so later `kill_all()` calls and the
548
+ exit do not wait for it again; its pid is named on stderr, unless quiet.
549
+ - Known limits (see [Process groups and signals](#process-groups-and-signals)): a Ctrl-C raised just as
550
+ a command's fork returns, before `Popen` has stored the pid, still loses the command, as in 1.54
551
+ (serial runs only, some 60-150 µs per command). On Python 3.6-3.10 a second Ctrl-C right after a
552
+ first one can lose a command that is starting, or leave one that a call waits for registered but not
553
+ signalled; on 3.6-3.9 a second one at a few other moments can lose it too, and a first one at a few
554
+ instructions can leave it registered but not signalled. On every Python, two Ctrl-Cs at nearly the same
555
+ moment, the first cutting short the stop of a timed-out command or of a finished command's background
556
+ processes, can leave them not signalled: `kill_all()` still reaches them, and so does the exit hook, except
557
+ on Python 3.6-3.10 for a finished command's background processes, which can then outlive the program. On
558
+ Python 3.6-3.9 and 3.14 a Ctrl-C just as
559
+ `threading` takes one of its own locks can leave that lock held and hang the program as its threads end
560
+ (CPython's own).
561
+
435
562
  ## License
436
563
 
437
564
  GPLv3+ — see the package metadata. Authored by Yufei Pan (<pan@zopyr.us>).
@@ -45,8 +45,8 @@ You can also just drop `multiCMD.py` into your project and import it directly.
45
45
  ```bash
46
46
  $ mcmd -h
47
47
  usage: mcmd [-h] [-p] [-t SECONDS] [-m max_threads] [--sudo] [-q] [-n]
48
- [--shell] [-f FILE] [-a FILE] [--summary | --no-summary]
49
- [--no-prefix] [--no-color] [--pty | --no-pty] [-V]
48
+ [--shell] [-e NAME=VALUE] [-f FILE] [-a FILE] [--summary |
49
+ --no-summary] [--no-prefix] [--no-color] [--pty | --no-pty] [-V]
50
50
  [command ...]
51
51
 
52
52
  Run multiple commands in parallel
@@ -70,6 +70,8 @@ options:
70
70
  -n, --dry-run print each command, quoted, instead of running it
71
71
  --shell run each command through /bin/sh -c ( pipes, ;, > and
72
72
  globs work )
73
+ -e, --env NAME=VALUE set NAME to VALUE in the environment of every command
74
+ ( repeatable ); NAME alone removes it
73
75
  -f, --file FILE read commands from FILE, one per line ( - for stdin;
74
76
  blank lines and # comments are skipped )
75
77
  -a, --arg-file FILE run each command template once per line of FILE, with
@@ -267,10 +269,47 @@ multiCMD.run_commands(cmds, max_threads=8, color=False) # never color
267
269
  multiCMD.run_command(["python3", "train.py"], pty=True) # live output, stderr merged
268
270
  ```
269
271
 
272
+ ### Environment per call
273
+
274
+ ```python
275
+ multiCMD.run_command(["xfs_info", "/data"], extra_env={"LC_ALL": "C"}) # add / override variables
276
+ multiCMD.run_command(["env"], extra_env={"HTTP_PROXY": None}) # None removes one
277
+ multiCMD.run_command(["env"], env={"PATH": "/usr/bin", "HOME": "/tmp"}) # the whole environment, as subprocess's env
278
+ ```
279
+
280
+ `env` is the whole environment of the command, as `subprocess`'s `env` ( names and values
281
+ are `str`; on POSIX `bytes` and `os.PathLike` work too ). `extra_env` is applied on top of the
282
+ inherited environment, or of `env` when both are given; a `None` value removes the variable,
283
+ and an empty `extra_env` is the same as none. "Inherited" is `os.environ` as the process holds
284
+ it when the command starts: a variable set behind Python's back ( `os.putenv`, `os.unsetenv`, a
285
+ C library's `setenv` ) reaches commands started without `env` / `extra_env`, not the ones
286
+ given them. An inherited variable with an empty name ( a corrupt `environ` entry such as `=x` ) is left out of the environment of a command given `env` / `extra_env`, and out of an `env` copied from `os.environ`; an empty name in `extra_env` is a `ValueError`. Names are case-insensitive on Windows, so `extra_env={"Path": ...}` replaces `PATH`
287
+ there. On POSIX a command is looked up in the `PATH` of the environment it gets, on every Python
288
+ version; Windows searches the calling process's `PATH` whatever the command's environment says.
289
+ Before Python 3.11 the launcher starts in this process's environment and is handed the command's on
290
+ top of it, so the two together must fit the system's argument limit ( `getconf ARG_MAX`; 2 MiB on Linux
291
+ with the default stack, 128 KiB for one string: before Python 3.11 an entry travels as `_MULTICMD_ENV_<i>=NAME=VALUE`, 15 bytes
292
+ plus the digits of `i` more than its `NAME=VALUE` ); one that does not fit fails with return
293
+ code 126, `Argument list too long`, on every Python.
294
+ On Python 3.7-3.10 a command started with
295
+ neither `env` nor `extra_env` still gets the `LC_CTYPE=C.UTF-8` that interpreter's locale coercion
296
+ sets in a C locale ( as before ); a command given one of them gets exactly the environment described.
297
+ Both are per command: they never change `os.environ` and never reach another command, so
298
+ concurrent commands each see their own. `AsyncExecutor` takes them too, as defaults a call
299
+ overrides. `sudo` resets the environment by default, so with `use_sudo` they may not reach the
300
+ command. On the command line: `mcmd -e LC_ALL=C -e UNSET_ME "cmd"`; `-n` shows them as
301
+ `env LC_ALL=C cmd` ( `env -i ...` for a whole environment ).
302
+
270
303
  `raise_error=True` raises `multiCMD.CommandError`; its `.tasks` holds the failed `Task`
271
304
  objects and its message lists each one with its return code and last error line.
272
305
  `Task` also records `start_time`, `end_time` and `duration`. With `dry_run=True` nothing
273
306
  runs; each command is printed once, quoted so it can be pasted into a shell.
307
+ A command that is not found (return code 127), cannot be executed (126) or that multiCMD
308
+ itself could not run (-1, `task._error`) says why in its `task.stderr`; multiCMD also
309
+ prints a line saying so on stderr, unless `quiet=True`. Before Python 3.11 the exec of a
310
+ command given by its path, or of a file found in `PATH` that may not be executed, happens in
311
+ the launcher's process: it exits 127 / 126 with the reason as the command's own stderr, which
312
+ then shows with the command's output ( unless quiet ) instead of as a line of multiCMD's own.
274
313
 
275
314
  ### Asynchronous / fire-and-forget
276
315
 
@@ -283,7 +322,8 @@ tasks = multiCMD.run_commands(
283
322
  max_threads=2, quiet=True,
284
323
  wait_for_return=False, return_object=True,
285
324
  )
286
- # tasks[i].returncode is None until that command completes
325
+ # tasks[i].returncode is None until that command completes; tasks[i].thread.join(timeout)
326
+ # waits for one command ( a command whose thread could not start fails with -1, its thread already done )
287
327
 
288
328
  # Later, block until everything launched this way has finished:
289
329
  multiCMD.join_threads()
@@ -310,7 +350,8 @@ executor holds at most one thread per permit plus the dispatcher, however many
310
350
  commands are queued. A command's `task.thread` is `None` until it starts, and goes
311
351
  back to `None` once it has finished and the executor has joined its thread, so a
312
352
  large batch does not keep a `Thread` object per finished command. `stop()` skips
313
- commands that have not started yet.
353
+ commands that have not started yet, and so does a Ctrl-C ( or an error ) that interrupts
354
+ `run_command(s)`, for that call's commands.
314
355
 
315
356
  ### Range expansion from Python
316
357
 
@@ -354,11 +395,57 @@ every command inherits through any shell.
354
395
 
355
396
  Because the commands are not in your program's process group, signals sent to that
356
397
  group do not reach them. multiCMD handles this for you: a call waiting on the main
357
- thread that gets Ctrl-C stops its commands before re-raising, `mcmd` stops them on
398
+ thread that gets Ctrl-C, or the SystemExit of a SIGTERM handler's `sys.exit()`
399
+ (`join_threads()` and `AsyncExecutor.wait()` included), stops its commands, the queued ones
400
+ skipped, before re-raising, giving them up to 5 s to exit (a second Ctrl-C stops waiting
401
+ for them at once, a serial run's command still sent SIGTERM; one that lands while multiCMD stops them or forgets the threads that never ran ends
402
+ it too), `mcmd` stops them on
358
403
  SIGTERM / SIGHUP (unless it was started with them ignored, as under `nohup`), and any
359
- commands still running when the interpreter exits are killed. A program that handles
404
+ commands still running when the interpreter exits are killed. From then on a command queued
405
+ before the exit on one of multiCMD's threads does not start: it fails (return code -1, "not started:
406
+ the interpreter is exiting"), as the thread would be frozen with the interpreter and the command
407
+ outlive it; a command run from an `atexit` handler, a `weakref.finalize` callback or a `__del__`
408
+ after that (on the main thread) runs as usual. A program that handles
360
409
  SIGTERM itself, or that runs multiCMD from its own threads (where Ctrl-C never
361
- arrives), can call `multiCMD.kill_all()` from its handler.
410
+ arrives), can call `multiCMD.kill_all()` from its handler. A command still running a second
411
+ after SIGKILL (in uninterruptible sleep, or not yours to signal) is no longer tracked, so later
412
+ calls and the exit do not wait for it again; multiCMD names its pid on stderr, unless quiet.
413
+
414
+ A Ctrl-C (or a signal handler's exception, such as a SIGTERM handler's `sys.exit()`) that lands
415
+ while a command starts stops it like any other, from the moment `Popen` knows the child's pid.
416
+ One that lands while the command is being forked can leave it running unregistered, as in 1.54:
417
+ if Python raises it as the fork returns, before `Popen` has stored the pid, neither the call nor
418
+ `kill_all()` nor the exit hook knows the command, which runs on in its own process group. Python
419
+ raises those on the main thread only, so this concerns serial runs (`max_threads=1`); and the window
420
+ is the fork's own length, some 60-150 µs per command however long it then runs. That is the only place
421
+ one Ctrl-C can lose a command. Before Python 3.11 a second one can too: on Python 3.6-3.10 a second
422
+ Ctrl-C that lands while the first is still on its way out of the command's start, before multiCMD has
423
+ begun to deal with it, is raised before multiCMD has registered the command (one that lands so while a
424
+ call waits for a running command leaves it registered but not signalled: `kill_all()` and the exit hook
425
+ still reach it), and on 3.6-3.9, where Python raises a Ctrl-C at nearly any instruction, a second one at a
426
+ few other moments of that handling can do the same, and even a first one, at a few instructions between the
427
+ steps of a run, can leave a command registered but not signalled. From Python 3.11 no other window is known where
428
+ a Ctrl-C loses a command. On every Python, two Ctrl-Cs at nearly the same moment, the first cutting short the stop
429
+ of a command that timed out or of what a finished command left running with its output open, can leave that
430
+ not signalled: `kill_all()` still reaches it, and so does the exit hook, except on Python 3.6-3.10 for what a
431
+ finished command left running, which can then outlive the program.
432
+ On Python 3.6-3.9 and 3.14 a Ctrl-C that lands just as Python's `threading` takes one of its own
433
+ locks with `with` (as `Thread.start()` does, for a serial run's output readers too) can leave that
434
+ lock held, and the program then hangs as its threads end. That is CPython's own, and any threaded
435
+ program can meet it; Python 3.10-3.13 do not leave the lock held.
436
+
437
+ multiCMD never touches SIGINT's handler (`mcmd` sets its own SIGTERM / SIGHUP handlers, nothing
438
+ else). A Ctrl-C that lands while multiCMD does its bookkeeping on the rare paths (a thread that
439
+ cannot start, stopping commands after a Ctrl-C or a failure, cancelling a batch, recording a command
440
+ that cannot be run) makes that bookkeeping run again from the start, and the KeyboardInterrupt (or
441
+ whatever your own handler raised) is raised once it is complete, so it neither cuts that short nor gets
442
+ lost. Your handler, if you installed one, is called with every Ctrl-C as it lands, and C-level hooks on
443
+ SIGINT (`faulthandler.register`, `signal.siginterrupt`) stay as they are. Whatever your handler raises
444
+ while a call waits for its commands is handled as a Ctrl-C: the commands are stopped (a serial run's
445
+ current one recorded as stopped, the ones after it skipped) and the exception is raised on. multiCMD
446
+ tells it from a failure of its own by your handler's frame in its traceback, and takes anything raised
447
+ while it only waits for a running command as yours; a handler that uninstalls itself before it
448
+ raises is not seen elsewhere, and what it raises then while a command starts counts as a failure.
362
449
 
363
450
  ## Key functions and objects
364
451
 
@@ -366,12 +453,13 @@ arrives), can call `multiCMD.kill_all()` from its handler.
366
453
  run_command(command, timeout=0, max_threads=1, quiet=False, dry_run=False,
367
454
  with_stdErr=False, return_code_only=False, return_object=False,
368
455
  wait_for_return=True, sem=None, use_sudo=..., raise_error=False,
369
- prefix=False, color=None, pty=False, shell=False)
456
+ prefix=False, color=None, pty=False, shell=False, env=None, extra_env=None)
370
457
 
371
458
  run_commands(commands, timeout=0, max_threads=1, quiet=False, dry_run=False,
372
459
  with_stdErr=False, return_code_only=False, return_object=False,
373
460
  parse=False, wait_for_return=True, sem=None, use_sudo=...,
374
- raise_error=False, prefix=False, color=None, pty=False, shell=False)
461
+ raise_error=False, prefix=False, color=None, pty=False, shell=False,
462
+ env=None, extra_env=None)
375
463
 
376
464
  ping(hosts, timeout=1, max_threads=0, ...) # returns True/False reachability
377
465
  join_threads(threads=..., timeout=None) # join fire-and-forget threads
@@ -380,7 +468,8 @@ set_sudo(use_sudo) # enable/disable sudo globally
380
468
 
381
469
  class Task: # command, returncode, stdout, stderr, start_time, end_time, duration
382
470
  class CommandError: # raised by raise_error=True; .tasks = the failed Tasks
383
- class AsyncExecutor: # run_command(s), wait, join, stop, cleanup, get_results, get_return_codes
471
+ class AsyncExecutor: # run_command(s), wait, join, stop, cleanup, get_results, get_return_codes;
472
+ # takes the options above, env / extra_env as defaults a call overrides
384
473
  ```
385
474
 
386
475
  The module also bundles a few terminal/formatting helpers used internally and
@@ -406,6 +495,44 @@ pytest -m "not slow" # skip intentional sleep-heavy cases
406
495
 
407
496
  `test_expand_ranges_perf.py` is a manual performance microbenchmark and is not collected by pytest.
408
497
 
498
+ ## Upgrading to 1.55 from 1.54
499
+
500
+ - New: a per-call environment. `env=` (the whole environment, as `subprocess`'s) and `extra_env=`
501
+ (added on top; a `None` value removes a variable) on `run_command`, `run_commands` and
502
+ `AsyncExecutor` (constructor defaults, overridden per call), and `mcmd -e NAME=VALUE` / `-e NAME`;
503
+ see [Environment per call](#environment-per-call).
504
+ - `quiet=True` now hides the lines about a command that could not be run: `Command path not found`
505
+ (127), `Command cannot be executed` (126) and the error of a command multiCMD could not run (-1),
506
+ which 1.54 printed even when quiet, partly on stdout. The reasons are still in `task.stderr`. Warnings
507
+ about the run's setup (sudo not found or not usable, no pty) still print.
508
+ - A Ctrl-C in `join_threads()` stops the commands it waits for, skips the ones still queued, waits up
509
+ to 5 s for them and re-raises; 1.54 re-raised and left them running. A Ctrl-C that interrupts
510
+ `AsyncExecutor.run_command(s)` skips that call's commands.
511
+ - A SIGTERM handler's `sys.exit()`, or whatever your own signal handler raises while a call waits for
512
+ its commands, stops them and records them as stopped before it is raised on, as a Ctrl-C does; 1.54
513
+ did that only for `KeyboardInterrupt`.
514
+ - A Ctrl-C while a command starts, times out or is stopped, or while multiCMD reads the output that
515
+ a finished command's background processes still hold open, no longer leaves the command or those
516
+ processes running unknown to `kill_all()` and the exit hook, or known to them but never signalled.
517
+ A second Ctrl-C ends the wait for the commands at once; each is still sent SIGTERM, and `kill_all()`
518
+ and the exit hook finish the kill. The limits below are the exceptions.
519
+ - Once the interpreter exits, a command queued on one of multiCMD's threads that has not started fails
520
+ with -1 ("not started: the interpreter is exiting") instead of starting and outliving the program.
521
+ - A command still running a second after SIGKILL is no longer tracked, so later `kill_all()` calls and the
522
+ exit do not wait for it again; its pid is named on stderr, unless quiet.
523
+ - Known limits (see [Process groups and signals](#process-groups-and-signals)): a Ctrl-C raised just as
524
+ a command's fork returns, before `Popen` has stored the pid, still loses the command, as in 1.54
525
+ (serial runs only, some 60-150 µs per command). On Python 3.6-3.10 a second Ctrl-C right after a
526
+ first one can lose a command that is starting, or leave one that a call waits for registered but not
527
+ signalled; on 3.6-3.9 a second one at a few other moments can lose it too, and a first one at a few
528
+ instructions can leave it registered but not signalled. On every Python, two Ctrl-Cs at nearly the same
529
+ moment, the first cutting short the stop of a timed-out command or of a finished command's background
530
+ processes, can leave them not signalled: `kill_all()` still reaches them, and so does the exit hook, except
531
+ on Python 3.6-3.10 for a finished command's background processes, which can then outlive the program. On
532
+ Python 3.6-3.9 and 3.14 a Ctrl-C just as
533
+ `threading` takes one of its own locks can leave that lock held and hang the program as its threads end
534
+ (CPython's own).
535
+
409
536
  ## License
410
537
 
411
538
  GPLv3+ — see the package metadata. Authored by Yufei Pan (<pan@zopyr.us>).