shexec 2.2.0__tar.gz → 2.3.0__tar.gz

{shexec-2.2.0/shexec.egg-info → shexec-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.2.0
+Version: 2.3.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 

{shexec-2.2.0 → shexec-2.3.0}/README.rst

@@ -113,7 +113,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 

{shexec-2.2.0 → shexec-2.3.0}/shexec/__init__.py

@@ -14,7 +14,7 @@ Dependencies:
 
 from __future__ import annotations
 
-__version__: str = "2.2.0"
+__version__: str = "2.3.0"
 
 from shexec.models import (
     DirNode,

{shexec-2.2.0 → shexec-2.3.0}/shexec/models.py

@@ -81,6 +81,10 @@ class PluginSpec:
         minCpu: the CPU ChRIS reserves, in millicores, as ``1000m``.
         minGpu: GPUs required; 0 for none.
         defaults: flag defaults, ``exec`` above all; a derived plugin's usual command.
+        omit: engine flags this plugin does not offer, by destination name, as
+            ``("threads", "threadsSplit")``. Each is removed from the parser, so
+            neither ChRIS's description nor the manual shows it, and its value is
+            pinned to the engine's default, or to ``defaults`` when that names it.
         bannerArt: the ASCII art on the manual's first page.
         bannerTitle: the one-line title under it.
         bannerText: the banner prose; ``{version}`` in it is replaced.
@@ -101,6 +105,7 @@ class PluginSpec:
     minCpu: str = "1000m"
     minGpu: int = 0
     defaults: dict[str, Any] = field(default_factory=dict)
+    omit: tuple[str, ...] = ()
     bannerArt: str = DEFAULT_BANNER_ART
     bannerTitle: str = ""
     bannerText: str = ""

{shexec-2.2.0 → shexec-2.3.0}/shexec/plugin.py

@@ -12,6 +12,7 @@ the first call of it; pl-freesurfer is the second, with FreeSurfer in the image
 Key components:
     - plugin_build: the factory
     - synopsis_build: the manual text for a spec
+    - parser_omit, manualText_omit: take flags a plugin does not offer out of its parser and manual
     - PLUGIN_SIGNATURE: the signature chris_plugin inspects
 
 Typical usage:
@@ -30,7 +31,8 @@ from __future__ import annotations
 
 import argparse
 import inspect
-from collections.abc import Callable
+import re
+from collections.abc import Callable, Sequence
 from pathlib import Path
 
 from chris_plugin import chris_plugin
@@ -51,6 +53,47 @@ PLUGIN_SIGNATURE: inspect.Signature = inspect.Signature(
 )
 
 
+ENTRY_LINE: re.Pattern[str] = re.compile(r"^ {8}(?:\[--(?P<flag>[A-Za-z][\w-]*)|[\[<])")
+
+
+def manualText_omit(text: str, flags: Sequence[str]) -> str:
+    r"""The shared manual text without the entries for the given flags.
+
+    The shared sections write each flag in one of two forms, both indented eight
+    spaces: a synopsis line, ``[--threads <numThreads>]`` ending in a continuation
+    backslash, and an ARGS entry, that line followed by its prose up to the next
+    entry, which opens with ``[`` or ``<``, or the next section heading. Both are
+    removed. A synopsis line that loses its successor keeps no dangling backslash.
+
+    Args:
+        text: a block of the shared manual, ``CLI_DS`` or ``ARGS_SYNOPSIS_DS``.
+        flags: flag names without dashes.
+
+    Returns:
+        The block without those flags.
+
+    Example:
+        >>> manualText_omit("        [--a] \\\n        [--b]\n", ["b"])
+        '        [--a]\n'
+    """
+    if not flags:
+        return text
+    omitted: set[str] = set(flags)
+    kept: list[str] = []
+    skipping: bool = False
+    for line in text.splitlines(keepends=True):
+        match: re.Match[str] | None = ENTRY_LINE.match(line)
+        if match:
+            skipping = match.group("flag") in omitted
+        elif skipping and line.strip() and not line.startswith(" " * 8):
+            skipping = False
+        if not skipping:
+            kept.append(line)
+    if kept and text.rstrip().endswith("]") and kept[-1].rstrip().endswith("\\"):
+        kept[-1] = kept[-1].rstrip()[:-1].rstrip() + "\n"
+    return "".join(kept)
+
+
 def synopsis_build(spec: PluginSpec, shortOnly: bool = False) -> str:
     """The manual text for a plugin: its NAME, SYNOPSIS and brief example, then the shared sections.
 
@@ -70,13 +113,20 @@ def synopsis_build(spec: PluginSpec, shortOnly: bool = False) -> str:
     SYNOPSIS
 
         {spec.name} \\"""
-        + CLI_DS
+        + manualText_omit(CLI_DS, spec.omit)
         + "\n\n    BRIEF EXAMPLE\n"
         + spec.briefExample
     )
     if shortOnly:
         return shortSynopsis
-    return shortSynopsis + spec.description + "\n    ARGS " + ARGS_SYNOPSIS_DS + TAG_HELP + spec.examples
+    return (
+        shortSynopsis
+        + spec.description
+        + "\n    ARGS "
+        + manualText_omit(ARGS_SYNOPSIS_DS, spec.omit)
+        + TAG_HELP
+        + spec.examples
+    )
 
 
 def manual_build(spec: PluginSpec) -> Manual:
@@ -92,13 +142,57 @@ def manual_build(spec: PluginSpec) -> Manual:
     )
 
 
+def parser_omit(parser: argparse.ArgumentParser, names: Sequence[str]) -> None:
+    """Take flags out of a parser, keeping each one's default as a fixed value.
+
+    argparse has no public way to remove an argument, and the parser is assembled
+    from pftree's, pfdo's and pfdo_run's parents, so the flag is removed from the
+    three places argparse keeps it: the parser's actions, which chris_plugin
+    serialises; the group that lists it in ``--help``; and the option-string map
+    the command line is parsed against. Its default stays in the namespace, set
+    with ``set_defaults``, so the engine reads the value it always did.
+
+    Args:
+        parser: the assembled parser, changed in place.
+        names: destination names, as ``threads``.
+
+    Raises:
+        ValueError: a name the parser does not have; a misspelt omission would
+            otherwise leave the flag in place unnoticed.
+
+    Example:
+        >>> p = argparse.ArgumentParser(); _ = p.add_argument("--n", dest="n", default=3)
+        >>> parser_omit(p, ["n"]); p.parse_args([]).n
+        3
+    """
+    # argparse internals: _actions, _action_groups, _group_actions and
+    # _option_string_actions are private but stable from 3.11 to 3.14, and
+    # chris_plugin itself reads _actions; CI runs every one of those versions.
+    actions: dict[str, argparse.Action] = {action.dest: action for action in parser._actions}
+    unknown: list[str] = [name for name in names if name not in actions]
+    if unknown:
+        raise ValueError(f"cannot omit flags the parser does not have: {', '.join(unknown)}")
+    for name in names:
+        action: argparse.Action = actions[name]
+        parser._remove_action(action)
+        for group in parser._action_groups:
+            if action in group._group_actions:
+                group._group_actions.remove(action)
+        for optionString in action.option_strings:
+            parser._option_string_actions.pop(optionString, None)
+        parser.set_defaults(**{name: action.default})
+
+
 def parser_forSpec(spec: PluginSpec) -> argparse.ArgumentParser:
-    """pfdo_run's parser named for the plugin, its defaults set from the spec.
+    """pfdo_run's parser named for the plugin, its omitted flags removed, its defaults set from the spec.
 
     The defaults are set on the parser rather than applied afterwards, so the
     description ``chris_plugin_info`` serialises carries them and ChRIS shows them.
+    They are set after the omission, so a spec can pin an omitted flag to a value
+    of its own.
     """
     parser: argparse.ArgumentParser = parser_build(standalone=False, command=spec.name, version=spec.version)
+    parser_omit(parser, spec.omit)
     parser.set_defaults(**spec.defaults)
     return parser
 

{shexec-2.2.0 → shexec-2.3.0/shexec.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.2.0
+Version: 2.3.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 

{shexec-2.2.0 → shexec-2.3.0}/tests/test_plugin.py

@@ -90,3 +90,42 @@ def test_shexecItself_isBuiltByTheFactory() -> None:
     assert (
         SPEC.name == "shexec" and PLUGIN.parser.get_default("exec") == "" and PLUGIN.main.__name__ == "shexec"
     )
+
+
+def test_omittedFlags_absentFromParserDescriptionAndManual_valuePinned(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    plugin: Plugin = plugin_build(spec_make(omit=("threads", "threadsSplit")), singleton=False)
+    names: set[str] = {parameter["name"] for parameter in serialize(plugin.parser)}
+    assert "threads" not in names and "threadsSplit" not in names
+    assert "verbosity" in names
+    options: argparse.Namespace = plugin.parser.parse_args([])
+    assert options.threads == 0 and options.threadsSplit == 1
+    with pytest.raises(SystemExit) as raised:
+        plugin.parser.parse_args(["--threads", "4"])
+    assert raised.value.code == 2
+    capsys.readouterr()
+    assert plugin.main(plugin.parser.parse_args(["--man"]), Path("/tmp"), Path("/tmp")) == 0
+    manual: str = capsys.readouterr().out
+    assert "[--threads" not in manual and "[--threadsSplit" not in manual
+    assert "[--verbosity" in manual
+
+
+def test_omittedFlag_pinnedBySpecDefault() -> None:
+    plugin: Plugin = plugin_build(
+        spec_make(omit=("threads",), defaults={"exec": COUNT_LINES, "threads": 2}), singleton=False
+    )
+    assert plugin.parser.parse_args([]).threads == 2
+    assert "threads" not in {parameter["name"] for parameter in serialize(plugin.parser)}
+
+
+def test_omittedFlag_unknownName_refused() -> None:
+    with pytest.raises(ValueError, match="threadz"):
+        plugin_build(spec_make(omit=("threadz",)), singleton=False)
+
+
+def test_omittedFlags_derivedPluginStillRuns(tmp_path: Path) -> None:
+    inputDir, outputDir = tree_make(tmp_path)
+    plugin: Plugin = plugin_build(spec_make(omit=("threads", "threadsSplit")), singleton=False)
+    assert plugin.main(plugin.parser.parse_args(["--verbosity", "0"]), inputDir, outputDir) == 0
+    assert (outputDir / "sub-01" / "jobs.jsonl").exists()