hypershell 2.7.2__tar.gz → 2.8.0__tar.gz

{hypershell-2.7.2 → hypershell-2.8.0}/.gitignore

@@ -133,3 +133,13 @@ dmypy.json
 
 # PyCharm
 .idea
+
+# MacOS
+.DS_Store
+
+# Local-only security review notes (not for publication)
+.security/
+
+# Agent memory not committed to project
+AGENTS.md
+CLAUDE.md

{hypershell-2.7.2 → hypershell-2.8.0}/PKG-INFO

@@ -1,7 +1,11 @@
 Metadata-Version: 2.4
 Name: hypershell
-Version: 2.7.2
+Version: 2.8.0
 Summary: A cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue.
+Project-URL: Homepage, https://hypershell.org
+Project-URL: Documentation, https://hypershell.readthedocs.io
+Project-URL: Repository, https://github.com/hypershell/hypershell
+Project-URL: Issues, https://github.com/hypershell/hypershell/issues
 Author-email: Geoffrey Lentner <glentner@purdue.edu>
 License-Expression: Apache-2.0
 License-File: LICENSE
@@ -11,24 +15,32 @@ Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: MacOS
 Classifier: Operating System :: Microsoft :: Windows
 Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Utilities
-Requires-Python: >=3.9
+Requires-Python: >=3.11
 Requires-Dist: cmdkit[toml]>=2.7.7
+Requires-Dist: cryptography>=3.4
 Requires-Dist: paramiko>=3.4.0
+Requires-Dist: psutil>=7.0.0
 Requires-Dist: pyyaml>=6.0.1
 Requires-Dist: rich>=13.7.1
-Requires-Dist: sqlalchemy>=2.0.29
+Requires-Dist: sqlalchemy>=2.0.50
 Requires-Dist: tomlkit>=0.13.2
+Provides-Extra: cron
+Requires-Dist: croniter>=6.2.2; extra == 'cron'
 Provides-Extra: postgres
-Requires-Dist: psycopg2>=2.9.9; extra == 'postgres'
+Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
+Provides-Extra: postgres-c
+Requires-Dist: psycopg[c]>=3.1; extra == 'postgres-c'
+Provides-Extra: postgres-system
+Requires-Dist: psycopg>=3.1; extra == 'postgres-system'
 Provides-Extra: uuid7
-Requires-Dist: uuid-utils>=0.10.0; extra == 'uuid7'
+Requires-Dist: uuid-utils>=0.16.0; extra == 'uuid7'
+Provides-Extra: zstd
+Requires-Dist: zstandard>=0.25.0; extra == 'zstd'
 Description-Content-Type: text/x-rst
 
 HyperShell v2: Distributed Task Execution for HPC
@@ -70,18 +82,36 @@ scalable workflow automation tool for *many-task* scenarios.
 
 Built on Python and tested on Linux, macOS, and Windows.
 
-Several tools offer similar functionality but not all together in a single tool with
-the user ergonomics we provide. Novel design elements include but are not limited to
+Other tools may offer similar functionality in some places but not within a single tool and
+not with the flexibility, ergonomics, and scalability provided by HyperShell.
 
-* **Cross-platform:** run on any platform where Python runs. In fact, the server and
-  client can run on different platforms in the same cluster.
-* **Client-server:** workloads do not need to be monolithic. Run the server as a
-  stand-alone service with SQLite or Postgres as a persistent database and dynamically
-  scale clients as needed.
-* **Staggered launch:** At the largest scales (1000s of nodes, 100k+ of workers),
-  the launch process can be challenging. Come up gradually to balance the workload.
-* **Database in-the-loop:** run in-memory for quick, ad-hoc workloads. Otherwise,
-  include a database for persistence, recovery when restarting, and search.
+Design elements include but are not limited to:
+
+* | **Client-server:** Run the server in stand-alone mode with SQLite or PostgreSQL.
+  | Scale clients elastically as needed (even down to zero).
+* | **Cross-platform:** trivial to install, run on any platform where Python runs.
+  | Mix platforms within a running cluster (Server on Linux, Clients on Windows).
+* | **Staggered launch:** Come up gradually to balance the workload.
+  | Scale to 1000+ nodes, 250k+ workers without crashing the server.
+* | **Database in-the-loop:** persist task metadata across runs.
+  | Fault-tolerant by default. Automated retries. Task history.
+* | **User-defined tags:** annotate tasks with ``key``:``value`` tags.
+  | Manage catalogs of large collections of tasks with ease.
+
+
+Usage
+-----
+
+*HyperShell* is primarily a command-line program.
+Most users will operate the ``hs cluster`` command (``hsx`` for short) in a start-to-finish workflow scenario much
+like people tend to do with alternatives like ``xargs``, `GNU Parallel <https://gnu.org/software/parallel>`_,
+or HPC-specific tools like `ParaFly <https://parafly.sourceforge.net>`_ or
+`TaskFarmer <https://docs.nersc.gov/jobs/workflow/taskfarmer/>`_ (NERSC-only) or
+`Launcher <https://tacc.utexas.edu/research/tacc-research/launcher/>`_ (TACC).
+
+.. code-block:: shell
+
+    seq 1000000 | hsx -t 'echo {}' -N64 --ssh 'a[00-32].cluster' > task.out
 
 
 Documentation
@@ -93,14 +123,28 @@ comprehensive usage guide on the command line you can view the manual page with
 ``man hs``.
 
 
-Contributions
--------------
+Support and Contributions
+-------------------------
+
+Join the `Discord <https://discord.gg/wmv5gyUfkN>`_ server to post questions, discuss your project,
+share with the community, keep in touch with announcements and upcoming events!
+
+*HyperShell* is an open-source project developed on `GitHub <https://github.com/hypershell/hypershell>`_.
+If you find bugs or issues with the software please create an `Issue <https://github.com/hypershell/hypershell/issues>`_.
+Contributions are welcome in the form of `Pull requests <https://github.com/hypershell/hypershell/pulls>`_
+for bug fixes, documentation, and minor feature improvements.
 
-Contributions are welcome. If you find bugs or have questions, open an *Issue* here.
 We've added a Code of Conduct recently, adapted from the
 `Contributor Covenant <https://www.contributor-covenant.org/>`_, version 2.0.
 
 
+License
+-------
+
+*HyperShell* is released under the
+`Apache Software License (v2) <https://www.apache.org/licenses/LICENSE-2.0>`_.
+
+
 Citation
 --------
 

{hypershell-2.7.2 → hypershell-2.8.0}/README.rst

@@ -37,18 +37,36 @@ scalable workflow automation tool for *many-task* scenarios.
 
 Built on Python and tested on Linux, macOS, and Windows.
 
-Several tools offer similar functionality but not all together in a single tool with
-the user ergonomics we provide. Novel design elements include but are not limited to
+Other tools may offer similar functionality in some places but not within a single tool and
+not with the flexibility, ergonomics, and scalability provided by HyperShell.
 
-* **Cross-platform:** run on any platform where Python runs. In fact, the server and
-  client can run on different platforms in the same cluster.
-* **Client-server:** workloads do not need to be monolithic. Run the server as a
-  stand-alone service with SQLite or Postgres as a persistent database and dynamically
-  scale clients as needed.
-* **Staggered launch:** At the largest scales (1000s of nodes, 100k+ of workers),
-  the launch process can be challenging. Come up gradually to balance the workload.
-* **Database in-the-loop:** run in-memory for quick, ad-hoc workloads. Otherwise,
-  include a database for persistence, recovery when restarting, and search.
+Design elements include but are not limited to:
+
+* | **Client-server:** Run the server in stand-alone mode with SQLite or PostgreSQL.
+  | Scale clients elastically as needed (even down to zero).
+* | **Cross-platform:** trivial to install, run on any platform where Python runs.
+  | Mix platforms within a running cluster (Server on Linux, Clients on Windows).
+* | **Staggered launch:** Come up gradually to balance the workload.
+  | Scale to 1000+ nodes, 250k+ workers without crashing the server.
+* | **Database in-the-loop:** persist task metadata across runs.
+  | Fault-tolerant by default. Automated retries. Task history.
+* | **User-defined tags:** annotate tasks with ``key``:``value`` tags.
+  | Manage catalogs of large collections of tasks with ease.
+
+
+Usage
+-----
+
+*HyperShell* is primarily a command-line program.
+Most users will operate the ``hs cluster`` command (``hsx`` for short) in a start-to-finish workflow scenario much
+like people tend to do with alternatives like ``xargs``, `GNU Parallel <https://gnu.org/software/parallel>`_,
+or HPC-specific tools like `ParaFly <https://parafly.sourceforge.net>`_ or
+`TaskFarmer <https://docs.nersc.gov/jobs/workflow/taskfarmer/>`_ (NERSC-only) or
+`Launcher <https://tacc.utexas.edu/research/tacc-research/launcher/>`_ (TACC).
+
+.. code-block:: shell
+
+    seq 1000000 | hsx -t 'echo {}' -N64 --ssh 'a[00-32].cluster' > task.out
 
 
 Documentation
@@ -60,14 +78,28 @@ comprehensive usage guide on the command line you can view the manual page with
 ``man hs``.
 
 
-Contributions
--------------
+Support and Contributions
+-------------------------
+
+Join the `Discord <https://discord.gg/wmv5gyUfkN>`_ server to post questions, discuss your project,
+share with the community, keep in touch with announcements and upcoming events!
+
+*HyperShell* is an open-source project developed on `GitHub <https://github.com/hypershell/hypershell>`_.
+If you find bugs or issues with the software please create an `Issue <https://github.com/hypershell/hypershell/issues>`_.
+Contributions are welcome in the form of `Pull requests <https://github.com/hypershell/hypershell/pulls>`_
+for bug fixes, documentation, and minor feature improvements.
 
-Contributions are welcome. If you find bugs or have questions, open an *Issue* here.
 We've added a Code of Conduct recently, adapted from the
 `Contributor Covenant <https://www.contributor-covenant.org/>`_, version 2.0.
 
 
+License
+-------
+
+*HyperShell* is released under the
+`Apache Software License (v2) <https://www.apache.org/licenses/LICENSE-2.0>`_.
+
+
 Citation
 --------
 

{hypershell-2.7.2 → hypershell-2.8.0}/docs/_include/client_help.rst

@@ -1,8 +1,10 @@
 Options
 ^^^^^^^
 
-``-N``, ``--num-tasks`` *NUM*
-    Number of task executors (default: 1).
+``-N``, ``--num-threads`` *NUM*
+    Number of task executor threads (default: 1).
+
+    Set to 0 to auto-detect based on available CPU cores.
 
 ``-t``, ``--template`` *CMD*
     Command-line template pattern (default: "{}").
@@ -20,7 +22,7 @@ Options
     Using larger bundles is a good idea for large distributed workflows; specifically, it is best
     to coordinate bundle size with the number of executors in use by each client.
 
-    See also ``--num-tasks`` and ``--bundlewait``.
+    See also ``--num-threads`` and ``--bundlewait``.
 
 ``-w``, ``--bundlewait`` *SEC*
     Seconds to wait before flushing tasks (default: 5).
@@ -69,7 +71,7 @@ Options
 ``-e``, ``--errors`` *PATH*
     File path for task errors (default: <stderr>).
 
-``-c``, ``--capture``
+``--capture``
     Capture individual task <stdout> and <stderr>.
 
     By default, the `stdout` and `stderr` streams of all tasks are fused with that of the `client`
@@ -77,23 +79,52 @@ Options
     tasks need to manage their own output, you can specify a redirect as part of a ``--template``,
     or use ``--capture`` to capture these as ``.out`` and ``.err`` files.
 
-    These are stored local to the `client` under `<prefix>/lib/task/<uuid>.[out,err]`.
-    Task outputs can be automatically retrieved via SFTP, see *task* usage.
+    These are stored local to the `client` under `<site>/lib/task/<uuid>.[out,err]`.
+    Task outputs can be automatically retrieved via SFTP, see `hs info`.
 
     Mutually exclusive with both ``--output`` and ``--errors``.
 
-``--timeout`` *SEC*
+``-T``, ``--timeout`` *SEC*
     Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).
 
     This feature allows for gracefully scaling down a cluster when task throughput subsides.
 
 
-``--task-timeout`` *SEC*
+``-W``, ``--task-timeout`` *SEC*
     Task-level walltime limit (default: none).
 
     Executors will send a progression of SIGINT, SIGTERM, and SIGKILL.
     If the process still persists the executor itself will shutdown.
 
+``-C``, ``--client-cores`` *NUM*
+    Limit available cores for client (default: all cores).
+
+    Sets an upper bound on the number of CPU cores that the client can use.
+    This allows running multiple clients on the same node by partitioning resources.
+    The client will not execute tasks if doing so would exceed this limit.
+
+    See also ``--num-threads=0``.
+
+``-M``, ``--client-memory`` *SIZE*
+    Limit available memory for client (default: all memory).
+
+    Sets an upper bound on the amount of memory that the client can use.
+    Specify memory size with units (e.g., '16GB', '4096MB'). This allows running
+    multiple clients on the same node by partitioning resources.
+
+``--monitor``
+    Enable resource monitoring for tasks.
+
+    When enabled, the client will monitor CPU and memory usage of running tasks
+    and their child processes, reporting peak usage back to the server.
+    Time-series data is stored in CSV format alongside task outputs.
+
+    See also ``--capture``.
+
+``-R``, ``--ratelimit`` *NUM*
+    Maximum allowed tasks per second per client. (default: none).
+    Individual tasks are throttled if exceeding this average limit.
+
 ``-S``, ``--signalwait`` *SEC*
     Task-level signal escalation wait period in seconds (default: 10).
 
@@ -101,3 +132,5 @@ Options
     this interval in seconds before escalating to the next level of interrupt.
 
     See also ``--task-timeout``.
+
+.. include:: /_include/tls_cli_options.rst

{hypershell-2.7.2 → hypershell-2.8.0}/docs/_include/client_usage.rst

@@ -1,4 +1,6 @@
 ``hs`` ``client`` ``[-h]``
     ``[-N NUM]`` ``[-t TEMPLATE]`` ``[-b SIZE]`` ``[-w SEC]`` ``[-H ADDR]`` ``[-p PORT]``
     ``[-k KEY]`` ``[--capture | [-o PATH] [-e PATH]]`` ``[--no-confirm]``
-    ``[-d SEC]`` ``[-T SEC]`` ``[-W SEC]`` ``[-S SEC]``
+    ``[-d SEC]`` ``[-T SEC]`` ``[-W SEC]`` ``[-S SEC]`` ``[-R NUM]``
+    ``[-C NUM]`` ``[-M MEM]`` ``[--monitor]``
+    ``[--no-tls | [--tls-ca PATH] [--tls-cert PATH] [--tls-key PATH]]``

{hypershell-2.7.2 → hypershell-2.8.0}/docs/_include/cluster_help.rst

@@ -25,8 +25,10 @@ Modes
 Options
 ^^^^^^^
 
-``-N``, ``--num-tasks`` *NUM*
-    Number of task executors per client (default: 1).
+``-N``, ``--num-threads`` *NUM*
+    Number of executor threads per client (default: 1).
+
+    Set to 0 to auto-detect based on available CPU cores on each client.
 
     For example, ``-N4`` would create four workers, but ``-N4 --ssh 'cluster-a[00-01].xyz'``
     creates two clients and a total of eight workers.
@@ -41,7 +43,13 @@ Options
 
     See section on `templates`.
 
-``-p``, ``--port`` *NUM*
+``-H``, ``--bind`` *ADDR*
+    Bind address (default: localhost, or 0.0.0.0 for remote clients).
+
+    Sets the address the server binds to. When launching clients on remote hosts
+    (``--ssh``, ``--mpi``, or ``--launcher``), bind to *0.0.0.0* so they can connect.
+
+``-p``, ``--port`` *PORT*
     Port number (default: 50001).
 
     This is an arbitrary choice and simply must be an available port. The default option chosen
@@ -58,7 +66,7 @@ Options
     Using larger bundles is a good idea for large distributed workflows; specifically, it is best
     to coordinate bundle size with the number of executors in use by each client.
 
-    See also ``--num-tasks`` and ``--bundlewait``.
+    See also ``--num-threads`` and ``--bundlewait``.
 
 ``-w``, ``--bundlewait`` *SEC*
     Seconds to wait before flushing tasks (default: 5).
@@ -69,6 +77,14 @@ Options
 
     See also ``--bundlesize``.
 
+``-Q``, ``--poll`` *NUM*
+    Polling interval in seconds between database queries if no tasks are available (default: 5).
+
+    This controls how frequently the scheduler checks the database for new tasks when idle.
+    Lower values (e.g., 1-2 seconds) provide faster responsiveness but may increase database
+    query load. Higher values (e.g., 10-30 seconds) reduce query frequency, useful for
+    workflows with infrequent task submission.
+
 ``-r``, ``--max-retries`` *NUM*
     Auto-retry failed tasks (default: 0).
 
@@ -94,7 +110,7 @@ Options
 ``--initdb``
     Auto-initialize database.
 
-    If a database is configured for use with the workflow (e.g., Postgres), auto-initialize
+    If a database is configured for use with the workflow (e.g., PostgreSQL), auto-initialize
     tables if they don't already exist. This is a short-hand for pre-creating tables with the
     ``hs initdb`` command. This happens by default with SQLite databases.
 
@@ -162,7 +178,7 @@ Options
     This also has the effect of staggering the workload. If your tasks take on the order of 30
     minutes and you have 1000 nodes, choose ``--delay-start=-1800``.
 
-``-c``, ``--capture``
+``--capture``
     Capture individual task <stdout> and <stderr>.
 
     By default, the `stdout` and `stderr` streams of all tasks are fused with that of the `client`
@@ -170,8 +186,8 @@ Options
     tasks need to manage their own output, you can specify a redirect as part of a ``--template``,
     or use ``--capture`` to capture these as ``.out`` and ``.err`` files.
 
-    These are stored local to the `client`. Task outputs can be automatically retrieved via SFTP,
-    see *task* usage.
+    These are stored local to the `client` under `<prefix>/lib/task/<uuid>.[out,err]`.
+    Task outputs can be automatically retrieved via SFTP, see `info`.
 
 ``-o``, ``--output`` *PATH*
     File path for task outputs (default: <stdout>).
@@ -195,8 +211,7 @@ Options
 ``-T``, ``--timeout`` *SEC*
     Timeout in seconds for clients. Automatically shutdown if no tasks received (default: never).
 
-    This option is only valid for an ``--autoscaling`` cluster. This feature allows for gracefully
-    scaling down a cluster when task throughput subsides.
+    This feature allows for gracefully scaling down a cluster when task throughput subsides.
 
 ``-W``, ``--task-timeout`` *SEC*
     Task-level walltime limit (default: none).
@@ -204,6 +219,48 @@ Options
     Executors will send a progression of SIGINT, SIGTERM, and SIGKILL.
     If the process still persists the executor itself will shutdown.
 
+``-c``, ``--cores`` *NUM*
+    Number of CPU cores required per task (default: none).
+
+    When specified, tasks will only be executed when the required number of cores
+    is available. Used with resource-aware scheduling and backfill.
+
+``-m``, ``--memory`` *MEM*
+    Amount of memory required per task (default: none).
+
+    Specify memory size with units (e.g., '4GB', '512MB'). Tasks will only be
+    executed when the required memory is available. Used with resource-aware
+    scheduling and backfill.
+
+``-C``, ``--client-cores`` *NUM*
+    Limit available cores per client (default: all cores).
+
+    Sets an upper bound on the number of CPU cores that each client can use.
+    This allows running multiple clients on the same node by partitioning resources.
+    Clients will not execute tasks if doing so would exceed this limit.
+
+    See also ``--num-threads=0``.
+
+``-M``, ``--client-memory`` *MEM*
+    Limit available memory per client (default: all memory).
+
+    Sets an upper bound on the amount of memory that each client can use.
+    Specify memory size with units (e.g., '16GB', '4096MB'). This allows running
+    multiple clients on the same node by partitioning resources.
+
+``--monitor``
+    Enable resource monitoring for tasks.
+
+    When enabled, clients will monitor CPU and memory usage of running tasks
+    and their child processes, reporting peak usage back to the server.
+    Time-series data is stored in CSV format alongside task outputs.
+
+    See also ``--capture``.
+
+``-R``, ``--ratelimit`` *NUM*
+    Maximum allowed tasks per second per client. (default: none).
+    Individual tasks are throttled if exceeding this average limit.
+
 ``-S``, ``--signalwait`` *SEC*
     Task-level signal escalation wait period in seconds (default: 10).
 
@@ -240,10 +297,15 @@ Options
     This value expresses some multiple of the average task duration in seconds.
 
     The autoscaler periodically checks ``toc / (factor x avg_duration)``, where
-    ``toc`` is the estimated time of completion for all remaining tasks given current
-    throughput of active clients. This ratio is referred to as *task pressure*, and if
-    it exceeds 1, the pressure is considered *high* and we will add another client if
-    we are not already at the given ``--max-size`` of the cluster.
+    ``toc`` is the estimated time of completion for remaining tasks in the active
+    *task group* given current throughput of active clients. This ratio is referred to
+    as *task pressure*, and if it exceeds 1, the pressure is considered *high* and we
+    will add another client if we are not already at the given ``--max-size`` of the
+    cluster.
+
+    Task pressure is relative to the active task group; tasks waiting in later groups
+    do not contribute. The scheduler only distributes tasks within the active group,
+    so scaling out for a future backlog would only create idle clients.
 
     For example, if the average task length is 30 minutes, and we set ``--factor=2``, then if
     the estimated time of completion of remaining tasks given currently connected executors
@@ -279,9 +341,11 @@ Options
     Only valid with ``--autoscaling``.
 
 ``-Y``, ``--max-size`` *SIZE*
-    Maximum size of cluster (default: 2).
+    Maximum size of cluster (default: 1).
 
     For a *dynamic* autoscaling policy, this sets an upper limit on the number of launched
     clients. When this number is reached, scaling stops regardless of task pressure.
 
     Only valid with ``--autoscaling``.
+
+.. include:: /_include/tls_cli_options.rst

hypershell-2.8.0/docs/_include/cluster_usage.rst

@@ -0,0 +1,8 @@
+``hs cluster [-h]`` ``[FILE | --restart | --forever]``
+    ``[-N NUM]`` ``[-t CMD]`` ``[-b SIZE]`` ``[-w SEC]`` ``[-Q NUM]``
+    ``[-H ADDR]`` ``[-p PORT]`` ``[-r NUM [--eager]]`` ``[-f PATH]`` ``[--capture | [-o PATH] [-e PATH]]``
+    ``[--ssh [HOST... | --ssh-group NAME] [--env] | --mpi | --launcher=ARGS...]``
+    ``[--no-db [--no-confirm] | --initdb]`` ``[-d SEC]`` ``[-T SEC]`` ``[-W SEC]`` ``[-S SEC]`` ``[-R NUM]``
+    ``[-c NUM]`` ``[-m MEM]`` ``[-C NUM]`` ``[-M MEM]`` ``[--monitor]``
+    ``[--autoscaling [MODE] [-P SEC] [-F VALUE] [-I NUM] [-X NUM] [-Y NUM]]``
+    ``[--no-tls | [--tls-ca PATH] [--tls-cert PATH] [--tls-key PATH]]``