hypershell 2.6.5__tar.gz → 2.6.6__tar.gz

hypershell-2.6.6/.dockerignore

@@ -0,0 +1 @@
+.venv

hypershell-2.6.6/.gitignore

@@ -0,0 +1,135 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# VSCode project
+.vscode/
+
+# PyCharm
+.idea

hypershell-2.6.6/.readthedocs.yaml

@@ -0,0 +1,16 @@
+# readthedocs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
+  jobs:
+    post_install:
+      - pip install uv
+      - UV_PROJECT_ENVIRONMENT=$READTHEDOCS_VIRTUALENV_PATH uv sync --all-extras --group docs --link-mode=copy
+
+sphinx:
+   configuration: docs/conf.py

hypershell-2.6.6/CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+glentner@purdue.edu.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

hypershell-2.6.6/Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.13-slim
+
+LABEL version="2.6.6"
+LABEL authors="glentner@purdue.edu"
+LABEL org.opencontainers.image.source="https://github.com/hypershell/hypershell"
+LABEL org.opencontainers.image.description="HyperShell Base Image"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+
+RUN apt-get update && \
+    apt-get upgrade --yes && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN addgroup --gid 1001 --system app && \
+    adduser --no-create-home --shell /bin/false --disabled-password --uid 1001 --system --group app
+
+RUN mkdir -p /var/lib/hypershell
+
+WORKDIR /app
+COPY . .
+RUN pip install .
+
+ENV HYPERSHELL_LOGGING_LEVEL=TRACE \
+    HYPERSHELL_LOGGING_STYLE=SYSTEM
+
+USER app
+ENTRYPOINT ["hs", "server"]
+CMD ["--forever"]

{hypershell-2.6.5 → hypershell-2.6.6}/PKG-INFO

@@ -1,35 +1,31 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: hypershell
-Version: 2.6.5
+Version: 2.6.6
 Summary: A cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue.
-License: Apache-2.0
-Keywords: distributed-computing,command-line-tool,shell-scripting,high-performance-computing,high-throughput-computing
-Author: Geoffrey Lentner
-Author-email: glentner@purdue.edu
-Requires-Python: >=3.9,<4.0
+Author-email: Geoffrey Lentner <glentner@purdue.edu>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: command-line-tool,distributed-computing,high-performance-computing,high-throughput-computing,shell-scripting
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: MacOS
 Classifier: Operating System :: Microsoft :: Windows
 Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Requires-Dist: cmdkit[toml]>=2.7.7
+Requires-Dist: paramiko>=3.4.0
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: rich>=13.7.1
+Requires-Dist: sqlalchemy>=2.0.29
+Requires-Dist: tomlkit>=0.13.2
 Provides-Extra: postgres
-Requires-Dist: cmdkit[toml] (>=2.7.7,<3.0.0)
-Requires-Dist: paramiko (>=3.4.0,<4.0.0)
-Requires-Dist: psycopg2 (>=2.9.9,<3.0.0) ; extra == "postgres"
-Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
-Requires-Dist: rich (>=13.7.1,<14.0.0)
-Requires-Dist: sqlalchemy (>=2.0.29,<3.0.0)
-Requires-Dist: tomlkit (>=0.13.2,<0.14.0)
-Project-URL: Documentation, https://hypershell.readthedocs.io
-Project-URL: Homepage, https://hypershell.org
-Project-URL: Repository, https://github.com/hypershell/hypershell
+Requires-Dist: psycopg2>=2.9.9; extra == 'postgres'
 Description-Content-Type: text/x-rst
 
 HyperShell v2: Distributed Task Execution for HPC
@@ -114,4 +110,3 @@ If *HyperShell* has helped in your research please consider citing us.
         numpages = {3},
         series = {PEARC '22}
     }
-

hypershell-2.6.6/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

hypershell-2.6.6/docs/_include/citation.rst

@@ -0,0 +1,15 @@
+.. code-block:: bibtex
+
+    @inproceedings{lentner_2022,
+        author = {Lentner, Geoffrey and Gorenstein, Lev},
+        title = {HyperShell v2: Distributed Task Execution for HPC},
+        year = {2022},
+        isbn = {9781450391610},
+        publisher = {Association for Computing Machinery},
+        url = {https://doi.org/10.1145/3491418.3535138},
+        doi = {10.1145/3491418.3535138},
+        booktitle = {Practice and Experience in Advanced Research Computing},
+        articleno = {80},
+        numpages = {3},
+        series = {PEARC '22}
+    }

hypershell-2.6.6/docs/_include/client_desc.rst

@@ -0,0 +1,8 @@
+Launch client directly, run tasks in parallel.
+
+The client connects to the server and pulls bundles of tasks off the shared queue.
+These tasks are run locally by some number of a parallel task executors.
+
+The environment for tasks are the same as for the client. Standard output and error
+for tasks are forwarded to that of the client, unless ``--capture`` is used, in which
+these are directed to individual files for each task.

hypershell-2.6.6/docs/_include/client_help.rst

@@ -0,0 +1,103 @@
+Options
+^^^^^^^
+
+``-N``, ``--num-tasks`` *NUM*
+    Number of task executors (default: 1).
+
+``-t``, ``--template`` *CMD*
+    Command-line template pattern (default: "{}").
+
+    This is expanded by the client just before execution. With the default "{}" the input
+    command-line will be run verbatim. Specifying a template pattern allows for simple input
+    arguments (e.g., file paths) to be transformed into some common form; such as
+    ``-t './some_command.py {} >outputs/{/-}.out'``.
+
+    See section on `templates`.
+
+``-b``, ``--bundlesize`` *SIZE*
+    Size of task bundle (default: 1).
+
+    Using larger bundles is a good idea for large distributed workflows; specifically, it is best
+    to coordinate bundle size with the number of executors in use by each client.
+
+    See also ``--num-tasks`` and ``--bundlewait``.
+
+``-w``, ``--bundlewait`` *SEC*
+    Seconds to wait before flushing tasks (default: 5).
+
+    The `client` collector thread that accumulates finished task bundles to return to
+    the `server` will push out a bundle after this period of time regardless of whether
+    it has reached the preferred bundle size.
+
+    See also ``--bundlesize``.
+
+``-H``, ``--host`` *ADDR*
+    Hostname or IP address to connect with server (default: localhost).
+
+``-p``, ``--port`` *NUM*
+    Port number to connect with server (default: 50001).
+
+``-k``, ``--auth`` *KEY*
+    Cryptographic authentication key to connect with server (default: <not secure>).
+
+    The default *KEY* used by the server and client is not secure and only a place holder.
+    It is expected that the user choose a secure *KEY*. The `cluster` automatically generates
+    a secure one-time *KEY*.
+
+``-d``, ``--delay-start`` *SEC*
+    Delay time in seconds before connecting to server (default: 0).
+
+    At larger scales it can be advantageous to uniformly delay the client launch sequence.
+    Hundreds or thousands of clients connecting to the server all at once is a challenge.
+    Even if the server could handle the load, your task throughput would be unbalanced,
+    coming in waves.
+
+    Use ``--delay-start`` with a negative number to impose a uniform random delay up to the
+    magnitude specified (e.g., ``--delay-start=-600`` would delay the client up to ten minutes).
+    This also has the effect of staggering the workload. If your tasks take on the order of 30
+    minutes and you have 1000 nodes, choose ``--delay-start=-1800``.
+
+``--no-confirm``
+    Disable client confirmation of task bundle received.
+
+    To achieve even higher throughput at large scales, optionally disable confirmation
+    payloads from clients. Consider using this option when also using ``--no-db``.
+
+``-o``, ``--output`` *PATH*
+    File path for task outputs (default: <stdout>).
+
+``-e``, ``--errors`` *PATH*
+    File path for task errors (default: <stderr>).
+
+``-c``, ``--capture``
+    Capture individual task <stdout> and <stderr>.
+
+    By default, the `stdout` and `stderr` streams of all tasks are fused with that of the `client`
+    thread, and in turn the `cluster`. If tasks are producing output that needs to be isolated, the
+    tasks need to manage their own output, you can specify a redirect as part of a ``--template``,
+    or use ``--capture`` to capture these as ``.out`` and ``.err`` files.
+
+    These are stored local to the `client` under `<prefix>/lib/task/<uuid>.[out,err]`.
+    Task outputs can be automatically retrieved via SFTP, see *task* usage.
+
+    Mutually exclusive with both ``--output`` and ``--errors``.
+
+``--timeout`` *SEC*
+    Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).
+
+    This feature allows for gracefully scaling down a cluster when task throughput subsides.
+
+
+``--task-timeout`` *SEC*
+    Task-level walltime limit (default: none).
+
+    Executors will send a progression of SIGINT, SIGTERM, and SIGKILL.
+    If the process still persists the executor itself will shutdown.
+
+``-S``, ``--signalwait`` *SEC*
+    Task-level signal escalation wait period in seconds (default: 10).
+
+    When tasks fail to halt following an initial SIGINT, the program waits
+    this interval in seconds before escalating to the next level of interrupt.
+
+    See also ``--task-timeout``.

hypershell-2.6.6/docs/_include/client_usage.rst

@@ -0,0 +1,4 @@
+``hs`` ``client`` ``[-h]``
+    ``[-N NUM]`` ``[-t TEMPLATE]`` ``[-b SIZE]`` ``[-w SEC]`` ``[-H ADDR]`` ``[-p PORT]``
+    ``[-k KEY]`` ``[--capture | [-o PATH] [-e PATH]]`` ``[--no-confirm]``
+    ``[-d SEC]`` ``[-T SEC]`` ``[-W SEC]`` ``[-S SEC]``

hypershell-2.6.6/docs/_include/cluster_desc.rst

@@ -0,0 +1,15 @@
+Start cluster locally or with remote clients over *ssh* or a custom *launcher*.
+This mode should be the most common entry-point for general usage. It fully encompasses all of the
+different agents in the system in a concise workflow.
+
+The input source for tasks is file-like, either a local path, or from *stdin* if no argument is
+given. The command-line tasks are pulled in and either directly published to a distributed queue
+(see ``--no-db``) or committed to a database first before being scheduled later.
+
+For large, long running workflows, it might be a good idea to configure a database and run an
+initial ``submit`` job to populate the database, and then run the cluster with ``--restart`` and no
+input *FILE*. If the cluster is interrupted for whatever reason it can gracefully restart where it
+left off.
+
+Use ``--autoscaling`` with either *fixed* or *dynamic* to run a persistent, elastically scalable
+cluster using an external ``--launcher`` to bring up clients as needed.