shexec 2.3.0__tar.gz → 2.5.0__tar.gz

{shexec-2.3.0/shexec.egg-info → shexec-2.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.3.0
+Version: 2.5.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -18,9 +18,9 @@ Requires-Python: >=3.11
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 Requires-Dist: chris_plugin>=0.4.0
-Requires-Dist: pfdo>=6.2.0
-Requires-Dist: pfdo_run>=4.2.0
-Requires-Dist: hostlib>=1.2.0
+Requires-Dist: pfdo>=6.3.0
+Requires-Dist: pfdo_run>=4.3.0
+Requires-Dist: hostlib>=1.3.0
 Requires-Dist: pflog>=2.0.0
 Provides-Extra: dev
 Requires-Dist: pytest>=8; extra == "dev"
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=(...)`` removes them from its parser, its ChRIS description and its manual, and fixes their values. A flag that suits a shell but not ChRIS goes in ``chrisHidden`` instead: it stays on the command line and in the manual and is left out of the description ChRIS reads, so in ChRIS it keeps its default. ``jobMemory="16Gi"`` says how much memory one job needs at most; ``--threads N`` is then capped at the memory the process may use, its container's or Slurm job's limit or the machine's available memory as ``hostlib.memory_limit`` reads it, divided by that figure, and the cap is said on standard error. pl-freesurfer uses both: one ``recon-all`` at a time in a 32Gi ChRIS container, several at once on a workstation or a Slurm node with the memory for them. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -159,7 +159,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.3.0 → shexec-2.5.0}/README.rst

@@ -113,7 +113,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=(...)`` removes them from its parser, its ChRIS description and its manual, and fixes their values. A flag that suits a shell but not ChRIS goes in ``chrisHidden`` instead: it stays on the command line and in the manual and is left out of the description ChRIS reads, so in ChRIS it keeps its default. ``jobMemory="16Gi"`` says how much memory one job needs at most; ``--threads N`` is then capped at the memory the process may use, its container's or Slurm job's limit or the machine's available memory as ``hostlib.memory_limit`` reads it, divided by that figure, and the cap is said on standard error. pl-freesurfer uses both: one ``recon-all`` at a time in a 32Gi ChRIS container, several at once on a workstation or a Slurm node with the memory for them. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -130,7 +130,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.3.0 → shexec-2.5.0}/pyproject.toml

@@ -13,7 +13,7 @@ authors = [{ email = "dev@babyMRI.org" }]
 # fields, which the [project] table does not emit; setup.cfg supplies them, and the
 # author name lives there alone so Author-email stays a bare address.
 requires-python = ">=3.11"
-dependencies = ["chris_plugin>=0.4.0", "pfdo>=6.2.0", "pfdo_run>=4.2.0", "hostlib>=1.2.0", "pflog>=2.0.0"]
+dependencies = ["chris_plugin>=0.4.0", "pfdo>=6.3.0", "pfdo_run>=4.3.0", "hostlib>=1.3.0", "pflog>=2.0.0"]
 classifiers = [
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.11",

{shexec-2.3.0 → shexec-2.5.0}/shexec/__init__.py

@@ -14,7 +14,7 @@ Dependencies:
 
 from __future__ import annotations
 
-__version__: str = "2.3.0"
+__version__: str = "2.5.0"
 
 from shexec.models import (
     DirNode,
@@ -28,9 +28,10 @@ from shexec.models import (
     ShExecOptions,
     Span,
     StageResult,
+    ThreadsCap,
 )
 from shexec.plugin import plugin_build
-from shexec.shexec import ShExec, options_build, plugin_run
+from shexec.shexec import ShExec, options_build, plugin_run, threads_cap
 
 __all__: list[str] = [
     "DirNode",
@@ -45,8 +46,10 @@ __all__: list[str] = [
     "ShExecOptions",
     "Span",
     "StageResult",
+    "ThreadsCap",
     "__version__",
     "options_build",
     "plugin_build",
     "plugin_run",
+    "threads_cap",
 ]

{shexec-2.3.0 → shexec-2.5.0}/shexec/models.py

@@ -7,12 +7,14 @@ Key components:
     - ShExecOptions: pfdo_run's options, for the plugin
     - PluginSpec: what a plugin built on shexec says about itself
     - Plugin: what ``plugin_build`` returns
+    - ThreadsCap: how many jobs a run may hold at once, given the memory it has
     - DirNode, FileJob, JobResult, JobSpec, HostFacts, RunResult, StageResult,
       ProcessResult, TreeOptions, PfdoOptions, PfdoRunOptions: re-exported
     - Span: pflog's record of the plugin's run, re-exported
+    - MemoryLimit: hostlib's figure for the memory the process may use, re-exported
 
 Dependencies:
-    - pfdo_run.models, pflog.models
+    - pfdo_run.models, pflog.models, hostlib.models
 """
 
 from __future__ import annotations
@@ -22,6 +24,7 @@ from collections.abc import Callable
 from dataclasses import dataclass, field
 from typing import Any
 
+from hostlib.models import MemoryLimit
 from pfdo_run.models import (
     DirNode,
     FileJob,
@@ -85,6 +88,13 @@ class PluginSpec:
             ``("threads", "threadsSplit")``. Each is removed from the parser, so
             neither ChRIS's description nor the manual shows it, and its value is
             pinned to the engine's default, or to ``defaults`` when that names it.
+        chrisHidden: engine flags offered on the command line and in the manual
+            but left out of the description ChRIS reads, by destination name. ChRIS
+            passes only the parameters its description lists, so in ChRIS each
+            keeps its default; run from a shell, the flag works.
+        jobMemory: the most memory one job needs, as ``16Gi``; empty for no figure.
+            When set, ``--threads`` is capped at the memory the process may use
+            divided by it, so that the jobs held at once fit.
         bannerArt: the ASCII art on the manual's first page.
         bannerTitle: the one-line title under it.
         bannerText: the banner prose; ``{version}`` in it is replaced.
@@ -106,6 +116,8 @@ class PluginSpec:
     minGpu: int = 0
     defaults: dict[str, Any] = field(default_factory=dict)
     omit: tuple[str, ...] = ()
+    chrisHidden: tuple[str, ...] = ()
+    jobMemory: str = ""
     bannerArt: str = DEFAULT_BANNER_ART
     bannerTitle: str = ""
     bannerText: str = ""
@@ -121,7 +133,9 @@ class Plugin:
     Attributes:
         name: the plugin's name.
         spec: what it was built from.
-        parser: its flags, defaults applied; for ``chris_plugin`` and for tests.
+        parser: its flags, defaults applied: what the command line parses.
+        chrisParser: the parser ChRIS's description is made from, ``parser``
+            without the spec's ``chrisHidden`` flags.
         main: the decorated main, the console script's target.
         manual: its manual page.
 
@@ -133,10 +147,49 @@ class Plugin:
     name: str
     spec: PluginSpec
     parser: argparse.ArgumentParser
+    chrisParser: argparse.ArgumentParser
     main: Callable[..., int]
     manual: Any
 
 
+@dataclass
+class ThreadsCap:
+    """How many jobs a run may hold at once, and why.
+
+    Attributes:
+        requested: the ``--threads`` asked for; 0 is sequential.
+        allowed: what the run will use: ``requested``, or fewer when memory caps it.
+        jobBytes: the memory one job needs, in bytes; 0 when no figure was given.
+        limit: the memory the process may use, or ``None`` when it cannot be read.
+
+    Example:
+        >>> ThreadsCap(8, 3, 16 * 2**30, MemoryLimit(50 * 2**30, "cgroup")).capped()
+        True
+    """
+
+    requested: int
+    allowed: int
+    jobBytes: int
+    limit: MemoryLimit | None
+
+    def capped(self) -> bool:
+        """Whether memory lowered the request."""
+        return self.allowed < self.requested
+
+    def note(self) -> str:
+        """One line for standard error saying what was decided; empty when nothing needs saying."""
+        if self.requested <= 1 or self.jobBytes == 0:
+            return ""
+        jobGib: float = self.jobBytes / 2**30
+        if self.limit is None:
+            return f"--threads {self.requested}: memory limit unknown, not capped ({jobGib:g} GiB per job)"
+        have: str = f"{self.limit.gibibytes():.1f} GiB ({self.limit.source}) / {jobGib:g} GiB per job"
+        if self.capped():
+            fewer: str = "one at a time" if self.allowed == 0 else f"{self.allowed} at once"
+            return f"--threads {self.requested} capped to {fewer}: {have}"
+        return ""
+
+
 __all__: list[str] = [
     "DEFAULT_BANNER_ART",
     "DirNode",
@@ -144,6 +197,7 @@ __all__: list[str] = [
     "HostFacts",
     "JobResult",
     "JobSpec",
+    "MemoryLimit",
     "PfdoOptions",
     "PfdoRunOptions",
     "Plugin",
@@ -153,5 +207,6 @@ __all__: list[str] = [
     "ShExecOptions",
     "Span",
     "StageResult",
+    "ThreadsCap",
     "TreeOptions",
 ]

{shexec-2.3.0 → shexec-2.5.0}/shexec/plugin.py

@@ -30,6 +30,7 @@ Dependencies:
 from __future__ import annotations
 
 import argparse
+import copy
 import inspect
 import re
 from collections.abc import Callable, Sequence
@@ -38,7 +39,7 @@ from pathlib import Path
 from chris_plugin import chris_plugin
 from pfdo_run.__main__ import ARGS_SYNOPSIS_DS, CLI_DS, TAG_HELP, parser_build
 from pflog import timed
-from pftree.manual import Manual, manual_print
+from pftree.manual import Manual, manual_attach
 
 from shexec.models import Plugin, PluginSpec
 from shexec.shexec import plugin_run
@@ -189,17 +190,25 @@ def parser_forSpec(spec: PluginSpec) -> argparse.ArgumentParser:
     The defaults are set on the parser rather than applied afterwards, so the
     description ``chris_plugin_info`` serialises carries them and ChRIS shows them.
     They are set after the omission, so a spec can pin an omitted flag to a value
-    of its own.
+    of its own. ``--man`` and ``--synopsis`` print the plugin's own manual and exit as
+    they are read, so they need no ``inputdir outputdir``.
     """
     parser: argparse.ArgumentParser = parser_build(standalone=False, command=spec.name, version=spec.version)
     parser_omit(parser, spec.omit)
     parser.set_defaults(**spec.defaults)
+    manual_attach(parser, manual_build(spec))
     return parser
 
 
 def plugin_build(spec: PluginSpec, singleton: bool = True) -> Plugin:
     """Make a ChRIS plugin from a spec.
 
+    The command line is parsed by the spec's full parser. The description ChRIS
+    reads is made from a copy without the spec's ``chrisHidden`` flags: chris_plugin
+    describes whatever parser was registered, so the plugin registers the copy and
+    the console script parses with the original. ChRIS passes only the parameters
+    it was told of, so in ChRIS a hidden flag keeps its default.
+
     Args:
         spec: name, version, description for ChRIS, flag defaults, manual text.
         singleton: register with chris_plugin's registry, which ``chris_plugin_info``
@@ -214,25 +223,31 @@ def plugin_build(spec: PluginSpec, singleton: bool = True) -> Plugin:
         ('x', '')
     """
     parser: argparse.ArgumentParser = parser_forSpec(spec)
+    chrisParser: argparse.ArgumentParser = copy.deepcopy(parser)
+    parser_omit(chrisParser, spec.chrisHidden)
     manual: Manual = manual_build(spec)
 
     def body(options: argparse.Namespace, inputdir: Path, outputdir: Path) -> int:
-        if options.man or options.synopsis:
-            manual_print(manual, shortOnly=not options.man)
-            return 0
-        return plugin_run(options, inputdir, outputdir)
+        return plugin_run(options, inputdir, outputdir, jobMemory=spec.jobMemory, name=spec.name)
 
     body.__name__ = spec.name
     body.__qualname__ = spec.name
     body.__signature__ = PLUGIN_SIGNATURE  # type: ignore[attr-defined]  # what chris_plugin inspects; this module stringifies annotations
     timedBody: Callable[[argparse.Namespace, Path, Path], int] = timed(spec.name)(body)
-    main: Callable[..., int] = chris_plugin(
-        parser=parser,
-        title=spec.title,
-        category=spec.category,
-        min_memory_limit=spec.minMemory,
-        min_cpu_limit=spec.minCpu,
-        min_gpu_limit=spec.minGpu,
-        singleton=singleton,
-    )(timedBody)
-    return Plugin(name=spec.name, spec=spec, parser=parser, main=main, manual=manual)
+
+    def chris_wrap(using: argparse.ArgumentParser, register: bool) -> Callable[..., int]:
+        return chris_plugin(
+            parser=using,
+            title=spec.title,
+            category=spec.category,
+            min_memory_limit=spec.minMemory,
+            min_cpu_limit=spec.minCpu,
+            min_gpu_limit=spec.minGpu,
+            singleton=register,
+        )(timedBody)
+
+    main: Callable[..., int] = chris_wrap(parser, register=False)
+    if singleton:
+        # Registered for chris_plugin_info only; the console script is ``main``.
+        chris_wrap(chrisParser, register=True)
+    return Plugin(name=spec.name, spec=spec, parser=parser, chrisParser=chrisParser, main=main, manual=manual)

{shexec-2.3.0 → shexec-2.5.0}/shexec/shexec.py

@@ -8,13 +8,14 @@ configure logging, run, report, and return an exit status.
 Key components:
     - ShExec: pfdo_run's engine, built from the plugin's arguments
     - options_build: the plugin's arguments as ShExecOptions
+    - threads_cap: ``--threads`` against the memory the process may use
     - plugin_run: run and report, returning the exit status
 
 Typical usage:
     return plugin_run(options, inputdir, outputdir)
 
 Dependencies:
-    - pfdo_run, pftree.__main__ (logging), shexec.models
+    - pfdo_run, pftree.__main__ (logging), hostlib.facts, shexec.models
 """
 
 from __future__ import annotations
@@ -24,10 +25,11 @@ import json
 import sys
 from pathlib import Path
 
+from hostlib.facts import memory_limit, memorySize_parse
 from pfdo_run.pfdo_run import PfdoRun
 from pftree.__main__ import logging_configure
 
-from shexec.models import RunResult, ShExecOptions
+from shexec.models import MemoryLimit, RunResult, ShExecOptions, ThreadsCap
 
 EXIT_OK: int = 0
 EXIT_FAILED: int = 1
@@ -76,13 +78,41 @@ def result_report(options: argparse.Namespace, result: RunResult) -> None:
         print(f"Elapsed time = {result.runTime:.6f} seconds")
 
 
-def plugin_run(options: argparse.Namespace, inputdir: Path, outputdir: Path) -> int:
+def threads_cap(requested: int, jobBytes: int, limit: MemoryLimit | None) -> ThreadsCap:
+    """How many jobs may run at once when each needs ``jobBytes`` and the process has ``limit``.
+
+    No figure for a job, or no readable limit, leaves the request as it is. Otherwise
+    the run holds at most ``limit // jobBytes`` jobs; one or none fitting is 0, the
+    engine's sequential run, which is what one job at a time means.
+
+    Example:
+        >>> threads_cap(8, 16 * 2**30, MemoryLimit(50 * 2**30, "meminfo")).allowed
+        3
+        >>> threads_cap(4, 16 * 2**30, MemoryLimit(32 * 2**30, "cgroup")).allowed
+        2
+        >>> threads_cap(4, 16 * 2**30, MemoryLimit(20 * 2**30, "cgroup")).allowed
+        0
+    """
+    if requested <= 1 or jobBytes <= 0 or limit is None:
+        return ThreadsCap(requested=requested, allowed=requested, jobBytes=max(jobBytes, 0), limit=limit)
+    fitting: int = limit.bytes // jobBytes
+    allowed: int = requested if fitting >= requested else (fitting if fitting > 1 else 0)
+    return ThreadsCap(requested=requested, allowed=allowed, jobBytes=jobBytes, limit=limit)
+
+
+def plugin_run(
+    options: argparse.Namespace, inputdir: Path, outputdir: Path, jobMemory: str = "", name: str = "shexec"
+) -> int:
     """The body of the plugin's main.
 
     Args:
         options: the parsed flags; ``exec`` must be set.
         inputdir: the input tree.
         outputdir: the output tree.
+        jobMemory: the most memory one job needs, as ``16Gi``; when given,
+            ``--threads`` is capped so that the jobs held at once fit in the memory
+            the process may use, and the cap is said on standard error.
+        name: the command's name, which begins its messages on standard error.
 
     Returns:
         0 when at least one directory's jobs all succeeded (pftree's stage rule);
@@ -90,12 +120,17 @@ def plugin_run(options: argparse.Namespace, inputdir: Path, outputdir: Path) ->
         empty or a flag's value is unusable.
     """
     if not options.exec:
-        print("shexec: --exec is required; see shexec --man", file=sys.stderr)
+        print(f"{name}: --exec is required; see {name} --man", file=sys.stderr)
         return EXIT_USAGE
+    if jobMemory and getattr(options, "threads", 0) > 1:
+        cap: ThreadsCap = threads_cap(options.threads, memorySize_parse(jobMemory), memory_limit())
+        if cap.note():
+            print(f"{name}: {cap.note()}", file=sys.stderr)
+        options = argparse.Namespace(**{**vars(options), "threads": cap.allowed})
     try:
         engine: ShExec = ShExec.fromPlugin(options, inputdir, outputdir)
     except ValueError as err:
-        print(f"shexec: {err}", file=sys.stderr)
+        print(f"{name}: {err}", file=sys.stderr)
         return EXIT_USAGE
     logging_configure(engine.options.verbosity)
     result: RunResult = engine.run()

{shexec-2.3.0 → shexec-2.5.0/shexec.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shexec
-Version: 2.3.0
+Version: 2.5.0
 Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
 Home-page: https://github.com/FNNDSC/pl-shexec
 Author: FNNDSC
@@ -18,9 +18,9 @@ Requires-Python: >=3.11
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 Requires-Dist: chris_plugin>=0.4.0
-Requires-Dist: pfdo>=6.2.0
-Requires-Dist: pfdo_run>=4.2.0
-Requires-Dist: hostlib>=1.2.0
+Requires-Dist: pfdo>=6.3.0
+Requires-Dist: pfdo_run>=4.3.0
+Requires-Dist: hostlib>=1.3.0
 Requires-Dist: pflog>=2.0.0
 Provides-Extra: dev
 Requires-Dist: pytest>=8; extra == "dev"
@@ -142,7 +142,7 @@ pl-shexec is a library as well as a plugin, in the same way pfdo_run is a librar
     ))
     main = PLUGIN.main
 
-``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=(...)`` removes them from its parser, its ChRIS description and its manual, and fixes their values. A flag that suits a shell but not ChRIS goes in ``chrisHidden`` instead: it stays on the command line and in the manual and is left out of the description ChRIS reads, so in ChRIS it keeps its default. ``jobMemory="16Gi"`` says how much memory one job needs at most; ``--threads N`` is then capped at the memory the process may use, its container's or Slurm job's limit or the machine's available memory as ``hostlib.memory_limit`` reads it, divided by that figure, and the cap is said on standard error. pl-freesurfer uses both: one ``recon-all`` at a time in a 32Gi ChRIS container, several at once on a workstation or a Slurm node with the memory for them. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
 
 The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
 
@@ -159,7 +159,7 @@ From the command line, with the image:
         fnndsc/pl-shexec shexec --fileFilter txt                                \
         --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
 
-    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+    docker run --rm fnndsc/pl-shexec shexec --man
 
 With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
 

{shexec-2.3.0 → shexec-2.5.0}/shexec.egg-info/requires.txt

@@ -1,7 +1,7 @@
 chris_plugin>=0.4.0
-pfdo>=6.2.0
-pfdo_run>=4.2.0
-hostlib>=1.2.0
+pfdo>=6.3.0
+pfdo_run>=4.3.0
+hostlib>=1.3.0
 pflog>=2.0.0
 
 [dev]

{shexec-2.3.0 → shexec-2.5.0}/tests/test_cli.py

@@ -17,11 +17,15 @@ from shexec.plugin import synopsis_build
 
 
 def test_manual_andSynopsis(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
-    assert main(parser.parse_args(["--man"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        parser.parse_args(["--man"])
+    assert exc.value.code == 0
     out: str = capsys.readouterr().out
     body: str = out.split("DESCRIPTION", 1)[1]
     assert "`" not in body and "TAGS" in body and "<inputDir>" in body and "shexec" in body
-    assert main(parser.parse_args(["--synopsis"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        parser.parse_args(["--synopsis"])
+    assert exc.value.code == 0
     assert "TAGS" not in capsys.readouterr().out
 
 
@@ -62,3 +66,13 @@ def test_chris_plugin_info_describesThePlugin() -> None:
     assert description["documentation"] == "https://github.com/FNNDSC/pl-shexec"
     assert description["authors"] == "FNNDSC <dev@babyMRI.org>"
     assert any(p["name"] == "exec" for p in description["parameters"])
+
+
+def test_consoleScript_manNeedsNoDirectories(
+    monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    monkeypatch.setattr("sys.argv", ["shexec", "--man"])
+    with pytest.raises(SystemExit) as exc:
+        main()
+    assert exc.value.code == 0
+    assert "TAGS" in capsys.readouterr().out

{shexec-2.3.0 → shexec-2.5.0}/tests/test_plugin.py

@@ -11,7 +11,9 @@ import pytest
 from chris_plugin.parameters import serialize
 
 from shexec import Plugin, PluginSpec, plugin_build
+from shexec.models import MemoryLimit
 from shexec.plugin import PLUGIN_SIGNATURE
+from shexec.shexec import threads_cap
 
 COUNT_LINES: str = "wc -l %{inputWorkingDir}/%{inputWorkingFile}"
 
@@ -76,7 +78,9 @@ def test_derivedPlugin_describesItself_forChris() -> None:
 
 def test_derivedPlugin_manualAndVersion_areItsOwn(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
     plugin: Plugin = plugin_build(spec_make(), singleton=False)
-    assert plugin.main(plugin.parser.parse_args(["--man"]), tmp_path, tmp_path) == 0
+    with pytest.raises(SystemExit) as exc:
+        plugin.parser.parse_args(["--man"])
+    assert exc.value.code == 0
     out: str = capsys.readouterr().out
     assert "linecount" in out and "Counts lines." in out and "9.9.9" in out and "TAGS" in out
     with pytest.raises(SystemExit):
@@ -105,7 +109,8 @@ def test_omittedFlags_absentFromParserDescriptionAndManual_valuePinned(
         plugin.parser.parse_args(["--threads", "4"])
     assert raised.value.code == 2
     capsys.readouterr()
-    assert plugin.main(plugin.parser.parse_args(["--man"]), Path("/tmp"), Path("/tmp")) == 0
+    with pytest.raises(SystemExit):
+        plugin.parser.parse_args(["--man"])
     manual: str = capsys.readouterr().out
     assert "[--threads" not in manual and "[--threadsSplit" not in manual
     assert "[--verbosity" in manual
@@ -129,3 +134,48 @@ def test_omittedFlags_derivedPluginStillRuns(tmp_path: Path) -> None:
     plugin: Plugin = plugin_build(spec_make(omit=("threads", "threadsSplit")), singleton=False)
     assert plugin.main(plugin.parser.parse_args(["--verbosity", "0"]), inputDir, outputDir) == 0
     assert (outputDir / "sub-01" / "jobs.jsonl").exists()
+
+
+def test_chrisHidden_onCommandLineAndManual_notInDescription(capsys: pytest.CaptureFixture[str]) -> None:
+    plugin: Plugin = plugin_build(spec_make(chrisHidden=("threads", "threadsSplit")), singleton=False)
+    described: set[str] = {parameter["name"] for parameter in serialize(plugin.chrisParser)}
+    assert "threads" not in described and "threadsSplit" not in described and "verbosity" in described
+    assert "threads" in {parameter["name"] for parameter in serialize(plugin.parser)}
+    assert plugin.parser.parse_args(["--threads", "4"]).threads == 4
+    assert plugin.chrisParser.parse_args([]).threads == 0
+    with pytest.raises(SystemExit):
+        plugin.parser.parse_args(["--man"])
+    assert "[--threads" in capsys.readouterr().out
+
+
+def test_chrisHidden_unknownName_refused() -> None:
+    with pytest.raises(ValueError, match="thread"):
+        plugin_build(spec_make(chrisHidden=("thread",)), singleton=False)
+
+
+def test_jobMemory_capsThreads_andSaysSo(
+    tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
+) -> None:
+    inputDir, outputDir = tree_make(tmp_path)
+    (inputDir / "sub-02").mkdir()
+    (inputDir / "sub-02" / "t1.nii.gz").write_text("1\n")
+    monkeypatch.setattr("shexec.shexec.memory_limit", lambda: MemoryLimit(40 * 2**30, "cgroup"))
+    plugin: Plugin = plugin_build(spec_make(jobMemory="16Gi"), singleton=False)
+    options: argparse.Namespace = plugin.parser.parse_args(["--threads", "8", "--verbosity", "0"])
+    assert plugin.main(options, inputDir, outputDir) == 0
+    err: str = capsys.readouterr().err
+    assert "linecount: --threads 8 capped to 2 at once: 40.0 GiB (cgroup) / 16 GiB per job" in err
+    assert (outputDir / "sub-01" / "jobs.jsonl").exists() and (outputDir / "sub-02" / "jobs.jsonl").exists()
+
+
+def test_threadsCap_note() -> None:
+    gib: int = 2**30
+    assert (
+        threads_cap(4, 16 * gib, MemoryLimit(20 * gib, "meminfo"))
+        .note()
+        .startswith("--threads 4 capped to one at a time")
+    )
+    assert threads_cap(4, 16 * gib, MemoryLimit(80 * gib, "meminfo")).note() == ""
+    assert "not capped" in threads_cap(4, 16 * gib, None).note()
+    assert threads_cap(0, 16 * gib, MemoryLimit(1, "cgroup")).allowed == 0
+    assert threads_cap(4, 0, MemoryLimit(1, "cgroup")).allowed == 4