shexec 2.2.0__tar.gz

shexec-2.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 FNNDSC / BCH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

shexec-2.2.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: shexec
+Version: 2.2.0
+Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
+Home-page: https://github.com/FNNDSC/pl-shexec
+Author: FNNDSC
+Author-email: dev@babyMRI.org
+License: MIT
+Project-URL: Homepage, https://github.com/FNNDSC/pl-shexec
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX
+Requires-Python: >=3.11
+Description-Content-Type: text/x-rst
+License-File: LICENSE
+Requires-Dist: chris_plugin>=0.4.0
+Requires-Dist: pfdo>=6.2.0
+Requires-Dist: pfdo_run>=4.2.0
+Requires-Dist: hostlib>=1.2.0
+Requires-Dist: pflog>=2.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Dynamic: license-file
+
+pl-shexec
+=========
+
+**Run a command over every file of a tree, as a ChRIS plugin; and the pattern for turning any command-line program into one.**
+
+``pl-shexec`` is a `ChRIS <https://chrisproject.org/>`_ DS plugin that does two things. In a pipeline it is the plugin for the jobs too small to deserve one of their own: convert every image, zip every directory, rename, copy one file out of each subject, count lines. It takes a command template, runs it once per selected file of every directory in its input tree, and writes the results, and a record of each run, into an output tree that mirrors the input.
+
+.. code-block:: bash
+
+    shexec --fileFilter 'glob:*.jpg'                                            \
+        --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
+                        %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"      \
+        /incoming /outgoing
+
+It is also the design for wrapping a large existing program as a plugin. Put the program in the image, set the default command, and the plugin is done: the program's whole command line stays available through the template rather than a hand-picked subset of flags, and the tree walk, the records, the parallelism and the exit status come from the stack. With for example FreeSurfer in the image, one ``recon-all`` per subject directory is:
+
+.. code-block:: bash
+
+    shexec --analyzeFileIndex f --fileFilter 'glob:*.nii.gz'                     \
+        --exec "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile}
+                          -s %{inputWorkingDir|basename} -sd %{outputWorkingDir}" \
+        /incoming /outgoing
+
+pl-freesurfer is an example of the pattern: a plugin that depends on this one the way this one depends on pfdo_run, contributing the FreeSurfer image, that template as the default ``--exec``, and nothing else. See `Building a plugin on pl-shexec`_.
+
+The engine is `pfdo_run <https://github.com/FNNDSC/pfdo_run>`_; the plugin is pfdo_run with its directories supplied by ChRIS and its own run timed.
+
+The template
+------------
+
+``--exec`` is not a shell command. It is a template whose ``%{...}`` tags name the file and directories in hand, resolved once per file, and run directly without a shell, so a path with a space in it stays one argument and nothing is interpreted that was not written. Three tags describe the job:
+
+==========================  ==============================================
+``%{inputWorkingDir}``      the input directory being processed
+``%{outputWorkingDir}``     its counterpart in the output tree
+``%{inputWorkingFile}``     the file being processed
+==========================  ==============================================
+
+Functions derive one name from another: ``%{inputWorkingFile|rmext}.png`` turns ``scan.jpg`` into ``scan.png``; ``%{inputWorkingDir|basename}`` is the directory's own name. Every flag of the plugin is a tag too, and so are ``%{timestamp}``, ``%{hostname}`` and ``%{user}``. The grammar is `pftag <https://github.com/FNNDSC/pftag>`_'s; ``shexec --man`` lists it under TAGS. A template with no tag draws a warning, since the same command would then run for every file.
+
+What each run leaves behind
+---------------------------
+
+Each directory of the output tree gets, beside its results, one file, ``jobs.jsonl``, with one line per job that ran there:
+
+.. code-block:: text
+
+    scan.png
+    jobs.jsonl
+
+The line is the whole record of the job as JSON: the command as executed, everything it printed and everything it complained, its exit status, the host it ran on, and when it started and ended. ``pflog read out/**/jobs.jsonl`` prints a day's jobs as a table, ``--failed`` keeps the ones that went wrong, and a log collector indexes the lines by field.
+
+.. code-block:: text
+
+    STARTED                     DURATION  STATUS  NAME
+    2026-10-14T16:02:07-04:00      4.213  ok      convert /incoming/a/scan.jpg /outgoing/a/scan.png
+    2026-10-14T16:02:12-04:00      0.004  failed  convert /incoming/b/bad.jpg /outgoing/b/bad.png
+
+``--jobFiles`` also writes each job as four files beside its result, ``scan.jpg-cmd``, ``-stdout``, ``-stderr`` and ``-returncode``, the stream files rendered as a terminal would have shown them, for the reader who opens them in an editor. ``--noJobLogging`` keeps no record at all. ``--json`` prints the whole run's record on standard output.
+
+The plugin's own run is timed by `pflog <https://github.com/FNNDSC/pflog>`_: one line of JSON on standard error, when it began, when it ended, the seconds, and whether it succeeded. CUBE keeps that line with the rest of the plugin's output, and any log collector indexes it.
+
+Selecting files and directories
+-------------------------------
+
+The filters are pftree's. ``--fileFilter`` and ``--dirFilter`` take substrings, or ``glob:`` and ``re:`` patterns, joined by ``--fileFilterLogic`` and ``--dirFilterLogic``; ``--maxDepth`` bounds the descent. ``--analyzeFileIndex`` chooses which of a directory's selected files get the command: ``a`` for all, ``f`` for the first, ``l`` for the last, ``m`` for the middle, or an index. ``f`` is how one command runs per directory, started on the first file, as for a DICOM series.
+
+Directories run in parallel under ``--threads``. At ``--verbosity 2`` each command's output streams to the console as it runs.
+
+Exit status
+-----------
+
+0 when at least one directory's jobs all succeeded; 1 when every directory failed or the run could not start; 2 when ``--exec`` is missing or a flag's value is unusable. A failed template, a command that cannot be found, or a non-zero exit fails its directory and the run continues; the record says why.
+
+Wrapping a program
+------------------
+
+A plugin for a program with its own command line, FreeSurfer, ANTs, FSL, DCMTK, is pl-shexec with three changes and no new code:
+
+- the image carries the program, with whatever licence or data it needs;
+- the default ``--exec`` is the program's usual invocation as a template, so the common case needs no command at all and the unusual case overrides it;
+- the default ``--analyzeFileIndex`` chooses the unit, ``f`` for one run per directory, as for a subject or a DICOM series, ``a`` for one run per file.
+
+The user of such a plugin gets the program's entire interface, because anything the program accepts can be written into the template, and gets the same records, the same filters, the same threads and the same truthful exit status as every other plugin on the stack. This is the reason pl-shexec exists in the form it has.
+
+Building a plugin on pl-shexec
+------------------------------
+
+pl-shexec is a library as well as a plugin, in the same way pfdo_run is a library to pl-shexec. A downstream plugin does not copy it; it depends on the ``shexec`` package and calls ``plugin_build`` with a spec of what differs:
+
+.. code-block:: python
+
+    # freesurfer/__main__.py, the whole of pl-freesurfer's code
+    from shexec import PluginSpec, plugin_build
+
+    from freesurfer import __version__
+
+    RECON_ALL: str = (
+        "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile} "
+        "-s %{inputWorkingDir|basename} -sd %{outputWorkingDir}"
+    )
+
+    PLUGIN = plugin_build(PluginSpec(
+        name="freesurfer",
+        version=__version__,
+        title="FreeSurfer recon-all over every subject",
+        minMemory="8Gi",
+        minCpu="4000m",
+        defaults={"exec": RECON_ALL, "analyzeFileIndex": "f", "fileFilter": "glob:*.nii.gz"},
+        bannerTitle="FreeSurfer recon-all over an input tree",
+        description=DESCRIPTION,
+        examples=EXAMPLES,
+    ))
+    main = PLUGIN.main
+
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+
+The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
+
+Running it
+----------
+
+From ChRIS, add the plugin to a pipeline and give it ``--exec`` and filters as parameters.
+
+From the command line, with the image:
+
+.. code-block:: bash
+
+    docker run --rm -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw            \
+        fnndsc/pl-shexec shexec --fileFilter txt                                \
+        --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
+
+    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+
+With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
+
+As a Python package, ``pip install .`` in this repository gives the ``shexec`` command and the ``shexec`` module, whose ``plugin_run(options, inputdir, outputdir)`` is the body of the plugin and whose ``shexec.models`` re-exports every record pfdo_run, pfdo, pftree, hostlib and pflog produce.
+
+Development
+-----------
+
+.. code-block:: bash
+
+    uv venv --python /usr/bin/python3.14 .venv
+    uv pip install --python .venv/bin/python -e ".[dev]"
+    .venv/bin/ruff check . && .venv/bin/pytest -q
+    PATH=$PWD/.venv/bin:$PATH chris_plugin_info --dock-image docker.io/fnndsc/pl-shexec:2.1.0 shexec
+
+The CI workflow lints and tests on Python 3.11 to 3.14, builds the image on every push to ``main``, pushes it to Docker Hub and ghcr.io, and on a ``v*`` tag uploads the plugin's description to cube.chrisproject.org and publishes the ``shexec`` package to PyPI, so that a plugin built on pl-shexec can depend on it. The tag must match ``__version__`` in ``shexec/__init__.py``.
+
+Layout
+------
+
+.. code-block:: text
+
+    shexec/
+    ├── __main__.py   the plugin: parser, manual, @chris_plugin @timed main
+    ├── shexec.py     ShExec, options_build, plugin_run
+    └── models.py     ShExecOptions, and every record below re-exported
+
+``docs/purpose.adoc`` says why the plugin exists, ``docs/architecture.adoc`` how it is built, ``docs/history.adoc`` where it came from.

shexec-2.2.0/README.rst

@@ -0,0 +1,161 @@
+pl-shexec
+=========
+
+**Run a command over every file of a tree, as a ChRIS plugin; and the pattern for turning any command-line program into one.**
+
+``pl-shexec`` is a `ChRIS <https://chrisproject.org/>`_ DS plugin that does two things. In a pipeline it is the plugin for the jobs too small to deserve one of their own: convert every image, zip every directory, rename, copy one file out of each subject, count lines. It takes a command template, runs it once per selected file of every directory in its input tree, and writes the results, and a record of each run, into an output tree that mirrors the input.
+
+.. code-block:: bash
+
+    shexec --fileFilter 'glob:*.jpg'                                            \
+        --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
+                        %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"      \
+        /incoming /outgoing
+
+It is also the design for wrapping a large existing program as a plugin. Put the program in the image, set the default command, and the plugin is done: the program's whole command line stays available through the template rather than a hand-picked subset of flags, and the tree walk, the records, the parallelism and the exit status come from the stack. With for example FreeSurfer in the image, one ``recon-all`` per subject directory is:
+
+.. code-block:: bash
+
+    shexec --analyzeFileIndex f --fileFilter 'glob:*.nii.gz'                     \
+        --exec "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile}
+                          -s %{inputWorkingDir|basename} -sd %{outputWorkingDir}" \
+        /incoming /outgoing
+
+pl-freesurfer is an example of the pattern: a plugin that depends on this one the way this one depends on pfdo_run, contributing the FreeSurfer image, that template as the default ``--exec``, and nothing else. See `Building a plugin on pl-shexec`_.
+
+The engine is `pfdo_run <https://github.com/FNNDSC/pfdo_run>`_; the plugin is pfdo_run with its directories supplied by ChRIS and its own run timed.
+
+The template
+------------
+
+``--exec`` is not a shell command. It is a template whose ``%{...}`` tags name the file and directories in hand, resolved once per file, and run directly without a shell, so a path with a space in it stays one argument and nothing is interpreted that was not written. Three tags describe the job:
+
+==========================  ==============================================
+``%{inputWorkingDir}``      the input directory being processed
+``%{outputWorkingDir}``     its counterpart in the output tree
+``%{inputWorkingFile}``     the file being processed
+==========================  ==============================================
+
+Functions derive one name from another: ``%{inputWorkingFile|rmext}.png`` turns ``scan.jpg`` into ``scan.png``; ``%{inputWorkingDir|basename}`` is the directory's own name. Every flag of the plugin is a tag too, and so are ``%{timestamp}``, ``%{hostname}`` and ``%{user}``. The grammar is `pftag <https://github.com/FNNDSC/pftag>`_'s; ``shexec --man`` lists it under TAGS. A template with no tag draws a warning, since the same command would then run for every file.
+
+What each run leaves behind
+---------------------------
+
+Each directory of the output tree gets, beside its results, one file, ``jobs.jsonl``, with one line per job that ran there:
+
+.. code-block:: text
+
+    scan.png
+    jobs.jsonl
+
+The line is the whole record of the job as JSON: the command as executed, everything it printed and everything it complained, its exit status, the host it ran on, and when it started and ended. ``pflog read out/**/jobs.jsonl`` prints a day's jobs as a table, ``--failed`` keeps the ones that went wrong, and a log collector indexes the lines by field.
+
+.. code-block:: text
+
+    STARTED                     DURATION  STATUS  NAME
+    2026-10-14T16:02:07-04:00      4.213  ok      convert /incoming/a/scan.jpg /outgoing/a/scan.png
+    2026-10-14T16:02:12-04:00      0.004  failed  convert /incoming/b/bad.jpg /outgoing/b/bad.png
+
+``--jobFiles`` also writes each job as four files beside its result, ``scan.jpg-cmd``, ``-stdout``, ``-stderr`` and ``-returncode``, the stream files rendered as a terminal would have shown them, for the reader who opens them in an editor. ``--noJobLogging`` keeps no record at all. ``--json`` prints the whole run's record on standard output.
+
+The plugin's own run is timed by `pflog <https://github.com/FNNDSC/pflog>`_: one line of JSON on standard error, when it began, when it ended, the seconds, and whether it succeeded. CUBE keeps that line with the rest of the plugin's output, and any log collector indexes it.
+
+Selecting files and directories
+-------------------------------
+
+The filters are pftree's. ``--fileFilter`` and ``--dirFilter`` take substrings, or ``glob:`` and ``re:`` patterns, joined by ``--fileFilterLogic`` and ``--dirFilterLogic``; ``--maxDepth`` bounds the descent. ``--analyzeFileIndex`` chooses which of a directory's selected files get the command: ``a`` for all, ``f`` for the first, ``l`` for the last, ``m`` for the middle, or an index. ``f`` is how one command runs per directory, started on the first file, as for a DICOM series.
+
+Directories run in parallel under ``--threads``. At ``--verbosity 2`` each command's output streams to the console as it runs.
+
+Exit status
+-----------
+
+0 when at least one directory's jobs all succeeded; 1 when every directory failed or the run could not start; 2 when ``--exec`` is missing or a flag's value is unusable. A failed template, a command that cannot be found, or a non-zero exit fails its directory and the run continues; the record says why.
+
+Wrapping a program
+------------------
+
+A plugin for a program with its own command line, FreeSurfer, ANTs, FSL, DCMTK, is pl-shexec with three changes and no new code:
+
+- the image carries the program, with whatever licence or data it needs;
+- the default ``--exec`` is the program's usual invocation as a template, so the common case needs no command at all and the unusual case overrides it;
+- the default ``--analyzeFileIndex`` chooses the unit, ``f`` for one run per directory, as for a subject or a DICOM series, ``a`` for one run per file.
+
+The user of such a plugin gets the program's entire interface, because anything the program accepts can be written into the template, and gets the same records, the same filters, the same threads and the same truthful exit status as every other plugin on the stack. This is the reason pl-shexec exists in the form it has.
+
+Building a plugin on pl-shexec
+------------------------------
+
+pl-shexec is a library as well as a plugin, in the same way pfdo_run is a library to pl-shexec. A downstream plugin does not copy it; it depends on the ``shexec`` package and calls ``plugin_build`` with a spec of what differs:
+
+.. code-block:: python
+
+    # freesurfer/__main__.py, the whole of pl-freesurfer's code
+    from shexec import PluginSpec, plugin_build
+
+    from freesurfer import __version__
+
+    RECON_ALL: str = (
+        "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile} "
+        "-s %{inputWorkingDir|basename} -sd %{outputWorkingDir}"
+    )
+
+    PLUGIN = plugin_build(PluginSpec(
+        name="freesurfer",
+        version=__version__,
+        title="FreeSurfer recon-all over every subject",
+        minMemory="8Gi",
+        minCpu="4000m",
+        defaults={"exec": RECON_ALL, "analyzeFileIndex": "f", "fileFilter": "glob:*.nii.gz"},
+        bannerTitle="FreeSurfer recon-all over an input tree",
+        description=DESCRIPTION,
+        examples=EXAMPLES,
+    ))
+    main = PLUGIN.main
+
+``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.
+
+The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.
+
+Running it
+----------
+
+From ChRIS, add the plugin to a pipeline and give it ``--exec`` and filters as parameters.
+
+From the command line, with the image:
+
+.. code-block:: bash
+
+    docker run --rm -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw            \
+        fnndsc/pl-shexec shexec --fileFilter txt                                \
+        --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
+
+    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp
+
+With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.
+
+As a Python package, ``pip install .`` in this repository gives the ``shexec`` command and the ``shexec`` module, whose ``plugin_run(options, inputdir, outputdir)`` is the body of the plugin and whose ``shexec.models`` re-exports every record pfdo_run, pfdo, pftree, hostlib and pflog produce.
+
+Development
+-----------
+
+.. code-block:: bash
+
+    uv venv --python /usr/bin/python3.14 .venv
+    uv pip install --python .venv/bin/python -e ".[dev]"
+    .venv/bin/ruff check . && .venv/bin/pytest -q
+    PATH=$PWD/.venv/bin:$PATH chris_plugin_info --dock-image docker.io/fnndsc/pl-shexec:2.1.0 shexec
+
+The CI workflow lints and tests on Python 3.11 to 3.14, builds the image on every push to ``main``, pushes it to Docker Hub and ghcr.io, and on a ``v*`` tag uploads the plugin's description to cube.chrisproject.org and publishes the ``shexec`` package to PyPI, so that a plugin built on pl-shexec can depend on it. The tag must match ``__version__`` in ``shexec/__init__.py``.
+
+Layout
+------
+
+.. code-block:: text
+
+    shexec/
+    ├── __main__.py   the plugin: parser, manual, @chris_plugin @timed main
+    ├── shexec.py     ShExec, options_build, plugin_run
+    └── models.py     ShExecOptions, and every record below re-exported
+
+``docs/purpose.adoc`` says why the plugin exists, ``docs/architecture.adoc`` how it is built, ``docs/history.adoc`` where it came from.

shexec-2.2.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "shexec"
+dynamic = ["version"]
+description = "Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run"
+readme = "README.rst"
+license = { text = "MIT" }
+authors = [{ email = "dev@babyMRI.org" }]
+# chris_plugin_info reads the legacy Author, Author-email and Home-page metadata
+# fields, which the [project] table does not emit; setup.cfg supplies them, and the
+# author name lives there alone so Author-email stays a bare address.
+requires-python = ">=3.11"
+dependencies = ["chris_plugin>=0.4.0", "pfdo>=6.2.0", "pfdo_run>=4.2.0", "hostlib>=1.2.0", "pflog>=2.0.0"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: POSIX",
+]
+
+[project.urls]
+Homepage = "https://github.com/FNNDSC/pl-shexec"
+
+[project.scripts]
+shexec = "shexec.__main__:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "ruff>=0.6"]
+
+[tool.setuptools.packages.find]
+include = ["shexec*"]
+
+[tool.setuptools.dynamic]
+version = { attr = "shexec.__version__" }
+
+[tool.ruff]
+target-version = "py311"
+line-length = 110
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B", "C4", "SIM", "RUF"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

shexec-2.2.0/setup.cfg

@@ -0,0 +1,9 @@
+[metadata]
+author = FNNDSC
+author_email = dev@babyMRI.org
+url = https://github.com/FNNDSC/pl-shexec
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

shexec-2.2.0/shexec/__init__.py

@@ -0,0 +1,52 @@
+"""shexec: a ChRIS DS plugin that runs a shell-type command over every file of a tree.
+
+Key components:
+    - plugin_build, PluginSpec, Plugin: make a ChRIS plugin from a spec; what a
+      downstream plugin calls
+    - ShExec: pfdo_run's engine, built from the plugin's arguments
+    - ShExecOptions: the options
+    - plugin_run: the body of the plugin's main
+    - the records of pfdo_run and pflog, re-exported through shexec.models
+
+Dependencies:
+    - chris_plugin, pfdo_run, pflog
+"""
+
+from __future__ import annotations
+
+__version__: str = "2.2.0"
+
+from shexec.models import (
+    DirNode,
+    FileJob,
+    HostFacts,
+    JobResult,
+    JobSpec,
+    Plugin,
+    PluginSpec,
+    RunResult,
+    ShExecOptions,
+    Span,
+    StageResult,
+)
+from shexec.plugin import plugin_build
+from shexec.shexec import ShExec, options_build, plugin_run
+
+__all__: list[str] = [
+    "DirNode",
+    "FileJob",
+    "HostFacts",
+    "JobResult",
+    "JobSpec",
+    "Plugin",
+    "PluginSpec",
+    "RunResult",
+    "ShExec",
+    "ShExecOptions",
+    "Span",
+    "StageResult",
+    "__version__",
+    "options_build",
+    "plugin_build",
+    "plugin_run",
+]

shexec-2.2.0/shexec/__main__.py

@@ -0,0 +1,156 @@
+"""The ``shexec`` command: a ChRIS DS plugin that runs a command over every file of a tree.
+
+This module is shexec's own call of ``plugin_build``: its name, its description for
+ChRIS, no flag defaults, and its words on the manual page. A plugin built on shexec
+has a module of the same shape with its own spec; see ``shexec.plugin`` and the
+README's "Building a plugin on pl-shexec".
+
+Key components:
+    - SPEC: what shexec says about itself
+    - PLUGIN: the built plugin
+    - parser, main: the plugin's flags and the console script's target
+
+Typical usage:
+    shexec --fileFilter 'glob:*.jpg' --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
+        %{outputWorkingDir}/%{inputWorkingFile|rmext}.png" /incoming /outgoing
+
+Dependencies:
+    - shexec.plugin, shexec.models
+"""
+
+from __future__ import annotations
+
+import argparse
+from collections.abc import Callable
+
+from pftree.manual import Manual
+
+from shexec import __version__
+from shexec.models import Plugin, PluginSpec
+from shexec.plugin import plugin_build
+
+TITLE: str = "Execute shell-type commands across input spaces"
+CATEGORY: str = "Utility"
+MIN_MEMORY: str = "2Gi"
+MIN_CPU: str = "1000m"
+
+BANNER_ART: str = r"""
+       _            _
+      | |          | |
+ _ __ | |______ ___| |__   _____  _____  ___
+| '_ \| |______/ __| '_ \ / _ \ \/ / _ \/ __|
+| |_) | |      \__ \ | | |  __/>  <  __/ (__
+| .__/|_|      |___/_| |_|\___/_/\_\___|\___|
+| |
+|_|
+"""
+
+BANNER_TITLE: str = "Run shell-type commands over an input tree"
+
+BANNER_TEXT: str = """
+                    Run shell-type commands over an input tree
+
+                             -- version {version} --
+
+        A ChRIS DS plugin that runs one command for every selected file of
+        every directory in its input tree, the command written as a template
+        whose %{{...}} tags name the file and directories in hand. The output
+        tree mirrors the input tree; each command's record lands beside its
+        results. The engine is pfdo_run.
+"""
+
+DESCRIPTION: str = """
+    DESCRIPTION
+
+        ``shexec`` is pfdo_run as a ChRIS plugin. ChRIS gives it an input and
+        an output directory; every other flag is pfdo_run's. The input tree is
+        walked, files are selected in each directory by the filters and
+        --analyzeFileIndex, and the --exec template is run once per selected
+        file with its tags resolved against that directory and file, without
+        a shell. Each job's record, the command, its output, its exit status,
+        the host and the timing, is one line of JSON in jobs.jsonl in its
+        output directory unless --noJobLogging is given; --jobFiles adds the
+        four files beside the result; at --verbosity 2 a job's output is also
+        streamed to the console as it runs.
+
+        The plugin's own run is timed: one line of JSON on standard error, when
+        it began, when it ended, the seconds, and whether it succeeded. CUBE
+        keeps that line with the rest of the plugin's output.
+
+        The exit status is 0 when at least one directory's jobs all succeeded,
+        1 when every directory failed or the run could not start, 2 when --exec
+        is missing. The commands available are those of the container image:
+        the coreutils, zip and unzip, bc, perl, tar, ImageMagick's convert,
+        and the DCMTK tools.
+"""
+
+EXAMPLES: str = """
+
+    EXAMPLES
+
+    Convert every jpg in the input tree to png in the output tree:
+
+        shexec                                                                  \\
+            --fileFilter 'glob:*.jpg'                                           \\
+            --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
+                            %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"  \\
+            --threads 4 /incoming /outgoing
+
+    Each conversion's command, stdout, stderr, return code and timing are one
+    line in the directory's jobs.jsonl; ``pflog read /outgoing/**/jobs.jsonl``
+    lists them.
+
+    Copy one file out of each directory whose path names a subject:
+
+        shexec                                                                  \\
+            --dirFilter 100307                                                  \\
+            --exec "cp %{inputWorkingDir}/brain.mgz %{outputWorkingDir}/brain.mgz" \\
+            --analyzeFileIndex f --noJobLogging /incoming /outgoing
+
+    Zip each directory's files into one archive named after the directory:
+
+        shexec                                                                  \\
+            --analyzeFileIndex f                                                \\
+            --exec "zip -j %{outputWorkingDir}/%{inputWorkingDir|basename}.zip
+                           %{inputWorkingDir}/%{inputWorkingFile}"              \\
+            /incoming /outgoing
+
+    From the image, with the directories mounted:
+
+        docker run --rm -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw        \\
+            fnndsc/pl-shexec shexec --fileFilter txt                            \\
+            --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing
+"""
+
+
+BRIEF_EXAMPLE: str = """
+
+        shexec --fileFilter 'glob:*.jpg'                                        \\
+            --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
+                            %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"  \\
+            /incoming /outgoing
+"""
+
+SPEC: PluginSpec = PluginSpec(
+    name="shexec",
+    version=__version__,
+    title=TITLE,
+    category=CATEGORY,
+    minMemory=MIN_MEMORY,
+    minCpu=MIN_CPU,
+    bannerArt=BANNER_ART,
+    bannerTitle=BANNER_TITLE,
+    bannerText=BANNER_TEXT,
+    briefExample=BRIEF_EXAMPLE,
+    description=DESCRIPTION,
+    examples=EXAMPLES,
+)
+
+PLUGIN: Plugin = plugin_build(SPEC)
+parser: argparse.ArgumentParser = PLUGIN.parser
+main: Callable[..., int] = PLUGIN.main
+
+
+def manual_build() -> Manual:
+    """The shexec manual page."""
+    return PLUGIN.manual