shexec 2.2.0__tar.gz → 2.4.0__tar.gz

{shexec-2.2.0/shexec.egg-info → shexec-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.2.0
+Version: 2.4.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -18,8 +18,8 @@ Requires-Python: >=3.11
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 Requires-Dist: chris_plugin>=0.4.0
-Requires-Dist: pfdo>=6.2.0
-Requires-Dist: pfdo_run>=4.2.0
+Requires-Dist: pfdo>=6.3.0
+Requires-Dist: pfdo_run>=4.3.0
 Requires-Dist: hostlib>=1.2.0
 Requires-Dist: pflog>=2.0.0
 Provides-Extra: dev
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -159,7 +159,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.2.0 → shexec-2.4.0}/README.rst

@@ -113,7 +113,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -130,7 +130,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.2.0 → shexec-2.4.0}/pyproject.toml

@@ -13,7 +13,7 @@ authors = [{ email = "dev@babyMRI.org" }]
 # fields, which the [project] table does not emit; setup.cfg supplies them, and the
 # author name lives there alone so Author-email stays a bare address.
 requires-python = ">=3.11"
-dependencies = ["chris_plugin>=0.4.0", "pfdo>=6.2.0", "pfdo_run>=4.2.0", "hostlib>=1.2.0", "pflog>=2.0.0"]
+dependencies = ["chris_plugin>=0.4.0", "pfdo>=6.3.0", "pfdo_run>=4.3.0", "hostlib>=1.2.0", "pflog>=2.0.0"]
 classifiers = [
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.11",

{shexec-2.2.0 → shexec-2.4.0}/shexec/__init__.py

@@ -14,7 +14,7 @@ Dependencies:
 
 from __future__ import annotations
 
-__version__: str = "2.2.0"
+__version__: str = "2.4.0"
 
 from shexec.models import (
     DirNode,

{shexec-2.2.0 → shexec-2.4.0}/shexec/models.py

@@ -81,6 +81,10 @@ class PluginSpec:
         minCpu: the CPU ChRIS reserves, in millicores, as ``1000m``.
         minGpu: GPUs required; 0 for none.
         defaults: flag defaults, ``exec`` above all; a derived plugin's usual command.
+        omit: engine flags this plugin does not offer, by destination name, as
+            ``("threads", "threadsSplit")``. Each is removed from the parser, so
+            neither ChRIS's description nor the manual shows it, and its value is
+            pinned to the engine's default, or to ``defaults`` when that names it.
         bannerArt: the ASCII art on the manual's first page.
         bannerTitle: the one-line title under it.
         bannerText: the banner prose; ``{version}`` in it is replaced.
@@ -101,6 +105,7 @@ class PluginSpec:
     minCpu: str = "1000m"
     minGpu: int = 0
     defaults: dict[str, Any] = field(default_factory=dict)
+    omit: tuple[str, ...] = ()
     bannerArt: str = DEFAULT_BANNER_ART
     bannerTitle: str = ""
     bannerText: str = ""

{shexec-2.2.0 → shexec-2.4.0}/shexec/plugin.py

@@ -12,6 +12,7 @@ the first call of it; pl-freesurfer is the second, with FreeSurfer in the image
 Key components:
     - plugin_build: the factory
     - synopsis_build: the manual text for a spec
+    - parser_omit, manualText_omit: take flags a plugin does not offer out of its parser and manual
     - PLUGIN_SIGNATURE: the signature chris_plugin inspects
 
 Typical usage:
@@ -30,13 +31,14 @@ from __future__ import annotations
 
 import argparse
 import inspect
-from collections.abc import Callable
+import re
+from collections.abc import Callable, Sequence
 from pathlib import Path
 
 from chris_plugin import chris_plugin
 from pfdo_run.__main__ import ARGS_SYNOPSIS_DS, CLI_DS, TAG_HELP, parser_build
 from pflog import timed
-from pftree.manual import Manual, manual_print
+from pftree.manual import Manual, manual_attach
 
 from shexec.models import Plugin, PluginSpec
 from shexec.shexec import plugin_run
@@ -51,6 +53,47 @@ PLUGIN_SIGNATURE: inspect.Signature = inspect.Signature(
 )
 
 
+ENTRY_LINE: re.Pattern[str] = re.compile(r"^ {8}(?:\[--(?P<flag>[A-Za-z][\w-]*)|[\[<])")
+
+
+def manualText_omit(text: str, flags: Sequence[str]) -> str:
+    r"""The shared manual text without the entries for the given flags.
+
+    The shared sections write each flag in one of two forms, both indented eight
+    spaces: a synopsis line, ``[--threads <numThreads>]`` ending in a continuation
+    backslash, and an ARGS entry, that line followed by its prose up to the next
+    entry, which opens with ``[`` or ``<``, or the next section heading. Both are
+    removed. A synopsis line that loses its successor keeps no dangling backslash.
+
+    Args:
+        text: a block of the shared manual, ``CLI_DS`` or ``ARGS_SYNOPSIS_DS``.
+        flags: flag names without dashes.
+
+    Returns:
+        The block without those flags.
+
+    Example:
+        >>> manualText_omit("        [--a] \\\n        [--b]\n", ["b"])
+        '        [--a]\n'
+    """
+    if not flags:
+        return text
+    omitted: set[str] = set(flags)
+    kept: list[str] = []
+    skipping: bool = False
+    for line in text.splitlines(keepends=True):
+        match: re.Match[str] | None = ENTRY_LINE.match(line)
+        if match:
+            skipping = match.group("flag") in omitted
+        elif skipping and line.strip() and not line.startswith(" " * 8):
+            skipping = False
+        if not skipping:
+            kept.append(line)
+    if kept and text.rstrip().endswith("]") and kept[-1].rstrip().endswith("\\"):
+        kept[-1] = kept[-1].rstrip()[:-1].rstrip() + "\n"
+    return "".join(kept)
+
+
 def synopsis_build(spec: PluginSpec, shortOnly: bool = False) -> str:
     """The manual text for a plugin: its NAME, SYNOPSIS and brief example, then the shared sections.
 
@@ -70,13 +113,20 @@ def synopsis_build(spec: PluginSpec, shortOnly: bool = False) -> str:
     SYNOPSIS
 
         {spec.name} \\"""
-        + CLI_DS
+        + manualText_omit(CLI_DS, spec.omit)
         + "\n\n    BRIEF EXAMPLE\n"
         + spec.briefExample
     )
     if shortOnly:
         return shortSynopsis
-    return shortSynopsis + spec.description + "\n    ARGS " + ARGS_SYNOPSIS_DS + TAG_HELP + spec.examples
+    return (
+        shortSynopsis
+        + spec.description
+        + "\n    ARGS "
+        + manualText_omit(ARGS_SYNOPSIS_DS, spec.omit)
+        + TAG_HELP
+        + spec.examples
+    )
 
 
 def manual_build(spec: PluginSpec) -> Manual:
@@ -92,14 +142,60 @@ def manual_build(spec: PluginSpec) -> Manual:
     )
 
 
+def parser_omit(parser: argparse.ArgumentParser, names: Sequence[str]) -> None:
+    """Take flags out of a parser, keeping each one's default as a fixed value.
+
+    argparse has no public way to remove an argument, and the parser is assembled
+    from pftree's, pfdo's and pfdo_run's parents, so the flag is removed from the
+    three places argparse keeps it: the parser's actions, which chris_plugin
+    serialises; the group that lists it in ``--help``; and the option-string map
+    the command line is parsed against. Its default stays in the namespace, set
+    with ``set_defaults``, so the engine reads the value it always did.
+
+    Args:
+        parser: the assembled parser, changed in place.
+        names: destination names, as ``threads``.
+
+    Raises:
+        ValueError: a name the parser does not have; a misspelt omission would
+            otherwise leave the flag in place unnoticed.
+
+    Example:
+        >>> p = argparse.ArgumentParser(); _ = p.add_argument("--n", dest="n", default=3)
+        >>> parser_omit(p, ["n"]); p.parse_args([]).n
+        3
+    """
+    # argparse internals: _actions, _action_groups, _group_actions and
+    # _option_string_actions are private but stable from 3.11 to 3.14, and
+    # chris_plugin itself reads _actions; CI runs every one of those versions.
+    actions: dict[str, argparse.Action] = {action.dest: action for action in parser._actions}
+    unknown: list[str] = [name for name in names if name not in actions]
+    if unknown:
+        raise ValueError(f"cannot omit flags the parser does not have: {', '.join(unknown)}")
+    for name in names:
+        action: argparse.Action = actions[name]
+        parser._remove_action(action)
+        for group in parser._action_groups:
+            if action in group._group_actions:
+                group._group_actions.remove(action)
+        for optionString in action.option_strings:
+            parser._option_string_actions.pop(optionString, None)
+        parser.set_defaults(**{name: action.default})
+
+
 def parser_forSpec(spec: PluginSpec) -> argparse.ArgumentParser:
-    """pfdo_run's parser named for the plugin, its defaults set from the spec.
+    """pfdo_run's parser named for the plugin, its omitted flags removed, its defaults set from the spec.
 
     The defaults are set on the parser rather than applied afterwards, so the
     description ``chris_plugin_info`` serialises carries them and ChRIS shows them.
+    They are set after the omission, so a spec can pin an omitted flag to a value
+    of its own. ``--man`` and ``--synopsis`` print the plugin's own manual and exit as
+    they are read, so they need no ``inputdir outputdir``.
     """
     parser: argparse.ArgumentParser = parser_build(standalone=False, command=spec.name, version=spec.version)
+    parser_omit(parser, spec.omit)
     parser.set_defaults(**spec.defaults)
+    manual_attach(parser, manual_build(spec))
     return parser
 
 
@@ -123,9 +219,6 @@ def plugin_build(spec: PluginSpec, singleton: bool = True) -> Plugin:
     manual: Manual = manual_build(spec)
 
     def body(options: argparse.Namespace, inputdir: Path, outputdir: Path) -> int:
-        if options.man or options.synopsis:
-            manual_print(manual, shortOnly=not options.man)
-            return 0
         return plugin_run(options, inputdir, outputdir)
 
     body.__name__ = spec.name

{shexec-2.2.0 → shexec-2.4.0/shexec.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.2.0
+Version: 2.4.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -18,8 +18,8 @@ Requires-Python: >=3.11
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 Requires-Dist: chris_plugin>=0.4.0
-Requires-Dist: pfdo>=6.2.0
-Requires-Dist: pfdo_run>=4.2.0
+Requires-Dist: pfdo>=6.3.0
+Requires-Dist: pfdo_run>=4.3.0
 Requires-Dist: hostlib>=1.2.0
 Requires-Dist: pflog>=2.0.0
 Provides-Extra: dev
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -159,7 +159,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.2.0 → shexec-2.4.0}/shexec.egg-info/requires.txt

@@ -1,6 +1,6 @@
 chris_plugin>=0.4.0
-pfdo>=6.2.0
-pfdo_run>=4.2.0
+pfdo>=6.3.0
+pfdo_run>=4.3.0
 hostlib>=1.2.0
 pflog>=2.0.0
 

{shexec-2.2.0 → shexec-2.4.0}/tests/test_cli.py

@@ -17,11 +17,15 @@ from shexec.plugin import synopsis_build
 
 
 def test_manual_andSynopsis(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
-    assert main(parser.parse_args(["--man"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        parser.parse_args(["--man"])
+    assert exc.value.code == 0
     out: str = capsys.readouterr().out
     body: str = out.split("DESCRIPTION", 1)[1]
     assert "`" not in body and "TAGS" in body and "<inputDir>" in body and "shexec" in body
-    assert main(parser.parse_args(["--synopsis"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        parser.parse_args(["--synopsis"])
+    assert exc.value.code == 0
     assert "TAGS" not in capsys.readouterr().out
 
 
@@ -62,3 +66,13 @@ def test_chris_plugin_info_describesThePlugin() -> None:
     assert description["documentation"] == "https://github.com/FNNDSC/pl-shexec"
     assert description["authors"] == "FNNDSC <dev@babyMRI.org>"
     assert any(p["name"] == "exec" for p in description["parameters"])
+
+
+def test_consoleScript_manNeedsNoDirectories(
+    monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    monkeypatch.setattr("sys.argv", ["shexec", "--man"])
+    with pytest.raises(SystemExit) as exc:
+        main()
+    assert exc.value.code == 0
+    assert "TAGS" in capsys.readouterr().out

{shexec-2.2.0 → shexec-2.4.0}/tests/test_plugin.py

@@ -76,7 +76,9 @@ def test_derivedPlugin_describesItself_forChris() -> None:
 
 def test_derivedPlugin_manualAndVersion_areItsOwn(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
     plugin: Plugin = plugin_build(spec_make(), singleton=False)
-    assert plugin.main(plugin.parser.parse_args(["--man"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        plugin.parser.parse_args(["--man"])
+    assert exc.value.code == 0
     out: str = capsys.readouterr().out
     assert "linecount" in out and "Counts lines." in out and "9.9.9" in out and "TAGS" in out
     with pytest.raises(SystemExit):
@@ -90,3 +92,43 @@ def test_shexecItself_isBuiltByTheFactory() -> None:
     assert (
         SPEC.name == "shexec" and PLUGIN.parser.get_default("exec") == "" and PLUGIN.main.__name__ == "shexec"
     )
+
+
+def test_omittedFlags_absentFromParserDescriptionAndManual_valuePinned(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    plugin: Plugin = plugin_build(spec_make(omit=("threads", "threadsSplit")), singleton=False)
+    names: set[str] = {parameter["name"] for parameter in serialize(plugin.parser)}
+    assert "threads" not in names and "threadsSplit" not in names
+    assert "verbosity" in names
+    options: argparse.Namespace = plugin.parser.parse_args([])
+    assert options.threads == 0 and options.threadsSplit == 1
+    with pytest.raises(SystemExit) as raised:
+        plugin.parser.parse_args(["--threads", "4"])
+    assert raised.value.code == 2
+    capsys.readouterr()
+    with pytest.raises(SystemExit):
+        plugin.parser.parse_args(["--man"])
+    manual: str = capsys.readouterr().out
+    assert "[--threads" not in manual and "[--threadsSplit" not in manual
+    assert "[--verbosity" in manual
+
+
+def test_omittedFlag_pinnedBySpecDefault() -> None:
+    plugin: Plugin = plugin_build(
+        spec_make(omit=("threads",), defaults={"exec": COUNT_LINES, "threads": 2}), singleton=False
+    )
+    assert plugin.parser.parse_args([]).threads == 2
+    assert "threads" not in {parameter["name"] for parameter in serialize(plugin.parser)}
+
+
+def test_omittedFlag_unknownName_refused() -> None:
+    with pytest.raises(ValueError, match="threadz"):
+        plugin_build(spec_make(omit=("threadz",)), singleton=False)
+
+
+def test_omittedFlags_derivedPluginStillRuns(tmp_path: Path) -> None:
+    inputDir, outputDir = tree_make(tmp_path)
+    plugin: Plugin = plugin_build(spec_make(omit=("threads", "threadsSplit")), singleton=False)
+    assert plugin.main(plugin.parser.parse_args(["--verbosity", "0"]), inputDir, outputDir) == 0
+    assert (outputDir / "sub-01" / "jobs.jsonl").exists()