hypershell 2.8.2__tar.gz → 2.9.0a1__tar.gz

{hypershell-2.8.2 → hypershell-2.9.0a1}/.gitignore

@@ -139,7 +139,7 @@ dmypy.json
 
 # Local-only security review notes (not for publication)
 .security/
+.local/
 
-# Agent memory not committed to project
-AGENTS.md
-CLAUDE.md
+# Personal Claude Code settings (permission allowlist etc.); not for publication
+.agents/settings.local.json

{hypershell-2.8.2 → hypershell-2.9.0a1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypershell
-Version: 2.8.2
+Version: 2.9.0a1
 Summary: A cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue.
 Project-URL: Homepage, https://hypershell.org
 Project-URL: Documentation, https://hypershell.readthedocs.io

hypershell-2.9.0a1/docs/_include/cluster_desc.rst

@@ -0,0 +1,34 @@
+Start cluster locally or with remote clients over *ssh* or a custom *launcher*.
+This mode should be the most common entry-point for general usage. It fully encompasses all of the
+different agents in the system in a concise workflow.
+
+The input source for tasks is file-like, either a local path, or from *stdin* if no argument is
+given. The command-line tasks are pulled in and either directly published to a distributed queue
+(see ``--no-db``) or committed to a database first before being scheduled later.
+
+Alternatively, use ``--from-json`` to read tasks from a JSON file (``FILE[@path]``); each task
+object's keys become named ``{key}`` fields in the ``--template`` (and task tags). In this mode the
+template is expanded when tasks are ingested rather than by the clients. A JSON source participates
+in the same re-submission gating as a plain *FILE* — ``--restart`` (idempotent requeue),
+``--repeat``, and ``--update --restart`` all apply — keyed by the file's absolute path together
+with any ``@path`` selector.
+
+For large, long running workflows, it might be a good idea to configure a database and run an
+initial ``submit`` job to populate the database, and then run the cluster with ``--restart`` and no
+input *FILE*. If the cluster is interrupted for whatever reason it can gracefully restart where it
+left off.
+
+Alternatively, pass the *FILE* directly with ``--restart``: the submission is detected, only tasks
+that never landed are submitted, and interrupted tasks are re-run — so a requeued
+``hsx <FILE> --restart`` job is safe to run repeatedly. Use ``--repeat`` to deliberately submit a
+known file's tasks again as a new source, or ``--update --restart`` to add only the tasks that are
+new since the file was last seen.
+
+De-dup cost scales with the file's own source lineage rather than the whole database, so requeues stay
+fast at scale; for very large workflows, a one-time ``submit`` followed by a bare ``hsx --restart`` (no
+file re-read) is cheaper than re-feeding the *FILE* each run.
+
+Use ``--autoscaling`` with either *fixed* or *dynamic* to run a persistent, elastically scalable
+cluster using an external ``--launcher`` to bring up clients as needed.
+
+Use ``hsx`` in place of ``hs cluster``.

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/cluster_help.rst

@@ -43,6 +43,15 @@ Options
 
     See section on `templates`.
 
+``--from-json`` *SPEC*
+    Read tasks from a JSON file (``FILE[@path]``).
+
+    The optional dotted ``path`` after ``@`` locates the list of task objects inside the
+    document (e.g. ``plan.json@chunks``); with no ``@`` the file's top level must be the
+    list. Each task object's keys become named ``{key}`` fields in the ``--template`` and
+    task tags. In this mode the template is expanded when tasks are ingested (submit-side),
+    not by the clients.
+
 ``-H``, ``--bind`` *ADDR*
     Bind address (default: localhost, or 0.0.0.0 for remote clients).
 
@@ -132,17 +141,39 @@ Options
     Conflicts with ``--no-db`` and mutually exclusive to ``--restart``.
 
 ``--restart``
-    Start scheduling from last completed task.
-
-    Instead of pulling a new list of tasks from some input `FILE`, with ``--restart`` enabled the
-    `cluster` will restart scheduling tasks where it left off. Any task in the database that was
-    previously scheduled but not completed will be reverted.
-
-    For very large workflows, an effective strategy is to first use the ``submit`` workflow to
-    populate the database, and then to use ``--restart`` so that if the `cluster` is interrupted,
-    it can easily continue where it left off, halting if nothing to be done.
-
-    Conflicts with ``--no-db`` and mutually exclusive to ``--forever``.
+    Start scheduling from where a prior run left off.
+
+    With no *FILE*, ``--restart`` resumes purely from the database: any task previously
+    scheduled but not completed is reverted, scheduling continues, and the `cluster` halts when
+    nothing is left to do. This is the classic "populate with ``submit``, then requeue" strategy
+    for very large workflows — if the `cluster` is interrupted it can gracefully continue where
+    it left off.
+
+    When a *FILE* is given, ``--restart`` becomes file-aware and idempotent. The file is read
+    upfront (a content fingerprint and task count), and only tasks whose identity is not already
+    present from a prior submission of that same path are submitted; the revert-interrupted flow
+    re-runs anything that was mid-flight. Requeuing the same ``hsx <FILE> --restart`` job as many
+    times as needed therefore never double-submits work. If the file's content has changed since
+    it was last seen, the run is refused with a suggestion to use ``--update --restart``.
+
+    Conflicts with ``--no-db`` and ``--repeat``, and mutually exclusive to ``--forever``.
+
+``--repeat``
+    Re-submit a known file's tasks again as a new source.
+
+    Re-submitting a file that has already been ingested is normally refused. ``--repeat``
+    overrides that: the file is recorded as a brand-new source and *all* of its tasks are
+    submitted again, even if an identical prior submission exists. Combine with ``-t``/tags to
+    represent a new phase or trial of the same work.
+
+``--update``
+    Submit only the tasks that are new since a file was last seen.
+
+    Used together with ``--restart`` (``hsx <FILE> --update --restart``): the file is recorded as
+    a new source and only tasks whose identity is not already present in the prior same-path
+    lineage are submitted — novel tasks run while previously-submitted tasks are skipped. On its
+    own (without ``--restart``) it is ambiguous and refused; it also cannot be combined with
+    ``--repeat``.
 
 ``--ssh-args`` *ARGS*...
     Command-line arguments for SSH. For example, ``--ssh-args '-i ~/.ssh/my_key'``.

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/cluster_usage.rst

@@ -1,4 +1,4 @@
-``hs cluster [-h]`` ``[FILE | --restart | --forever]``
+``hs cluster [-h]`` ``[FILE | --from-json SPEC | --restart | --forever]`` ``[--repeat | --update]``
     ``[-N NUM]`` ``[-t CMD]`` ``[-b SIZE]`` ``[-w SEC]`` ``[-Q NUM]``
     ``[-H ADDR]`` ``[-p PORT]`` ``[-r NUM [--eager]]`` ``[-f PATH]`` ``[--capture | [-o PATH] [-e PATH]]``
     ``[--ssh [HOST... | --ssh-group NAME] [--env] | --mpi | --launcher=ARGS...]``

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/config_task_env.rst

@@ -57,6 +57,14 @@ A few common environment variables are defined for every task.
 ``TASK_ERRPATH``
     Absolute file path where standard error is directed (if defined).
 
+``TASK_SLOT``
+    Zero-based execution slot of the task's executor (``0 .. N-1`` for a client
+    running ``N`` tasks in parallel). Stable for the executor's lifetime; use it to
+    pin resources (cores, GPUs) without colliding with sibling tasks. Defaults to ``0``.
+
+``TASK_SLOT_COUNT``
+    Number of concurrent executors on the client (``N``). Defaults to ``1``.
+
 |
 
 Any user-defined tags are included as environment variables as well with

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/config_task_env_alt.rst

@@ -55,6 +55,14 @@ A few common environment variables are defined for every task.
 ``TASK_ERRPATH``
     Absolute file path where standard error is directed (if defined).
 
+``TASK_SLOT``
+    Zero-based execution slot of the task's executor (``0 .. N-1`` for a client
+    running ``N`` tasks in parallel). Stable for the executor's lifetime; use it to
+    pin resources (cores, GPUs) without colliding with sibling tasks. Defaults to ``0``.
+
+``TASK_SLOT_COUNT``
+    Number of concurrent executors on the client (``N``). Defaults to ``1``.
+
 Further, any environment variable starting with ``HYPERSHELL_EXPORT_`` will be injected
 into the task environment sans prefix; e.g., ``HYPERSHELL_EXPORT_FOO`` would define
 ``FOO`` in the task environment. You can also define such variables in the ``export``

hypershell-2.9.0a1/docs/_include/submit_desc.rst

@@ -0,0 +1,41 @@
+Submit one or more tasks.
+
+Submit one task as positional arguments.
+If a single positional argument refers to a non-executable file path,
+tasks will be read from the file, one per line (use ``-f``/``--task-file`` to be explicit).
+
+Tasks are accumulated and published in bundles to the database by default.
+With ``-q``/``--queue``, tasks are submitted directly to a live queue, bypassing the database.
+The ``-b``/``--bundlesize`` and ``-w``/``--bundlewait`` options control the
+size of these bundles and how long to wait before flushing tasks regardless of
+how many have accumulated.
+
+Pre-format tasks at submit time with template expansion using ``--template``.
+Any tags specified with ``-t``/``--tag`` are applied to all tasks submitted.
+
+Use the special comment syntax, ``# HYPERSHELL: ...``, to include resource limits or tags inline.
+If the comment is alone on the line it will be applied to all tasks that follow.
+
+With ``--from-json``, read tasks from a JSON file instead. The value is
+``FILE[@path]`` where the optional dotted ``path`` locates the list of task objects
+within the document (e.g. ``plan.json@chunks``); with no ``@`` the file's top level
+must itself be the list, and a ``FILE`` of ``-`` reads from ``<stdin>``. Each task
+object's keys become named ``{key}`` template fields (and task tags), and an optional
+``args`` key provides the base command (reachable as ``{}``). All ``{key}`` fields are
+validated against every task object up front, so submission is rejected before any task
+is committed if a field is missing.
+
+Re-submitting a named task file is guarded to prevent accidental double submission. Each
+named file is recorded as a *source* (its absolute path, a content fingerprint, and the
+task count); submitting the same file again with no flag is refused, and if the path was
+seen but the content changed the refusal suggests ``--update``. Pass ``--repeat`` to
+deliberately submit all tasks again as a new source, or ``--update`` to submit only tasks
+not already present from earlier versions of the same file. JSON task files (``--from-json``)
+are gated the same way, keyed by their absolute path together with any ``@path`` selector.
+Single-command and ``<stdin>`` submissions (including ``--from-json -``) are treated as
+explicit intent and are exempt from these checks.
+
+Detection and ``--update`` de-dup are index-backed on the source lineage, so their cost scales with
+that file's own size, not the size of the database: re-submitting a normal file stays fast even against
+a very large database, while ``--update`` on a source of millions of tasks pays to re-read and
+fingerprint the entire file to find what is new.

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/submit_help.rst

@@ -10,6 +10,33 @@ Options
 ``-f``, ``--task-file`` *FILE*
     Input file containing one task per line.
 
+``--from-json`` *SPEC*
+    Read tasks from a JSON file (``FILE[@path]``).
+
+    The optional dotted ``path`` after ``@`` locates the list of task objects inside
+    the document (e.g. ``plan.json@chunks``); with no ``@`` the file's top level must
+    be the list, and a ``FILE`` of ``-`` reads from ``<stdin>``. Each task object's keys
+    become named ``{key}`` template fields and task tags; an optional ``args`` key is the
+    base command (reachable as ``{}``). Every ``{key}`` is validated against all task
+    objects before anything is submitted.
+
+``--repeat``
+    Re-submit a known file's tasks again as a brand-new source.
+
+    By default, submitting a named file whose path and content match a prior submission is
+    refused to guard against accidental double submission. ``--repeat`` overrides that: the
+    file is ingested as a new source and all of its tasks are submitted again, even if
+    identical tasks already exist. May be combined with new ``-t``/``--tag`` values to mark
+    a new phase or trial.
+
+``--update``
+    Submit only the tasks in a known file that are not already present.
+
+    Creates a new source record and submits only tasks whose identity (a stable hash of the
+    pre-template arguments, group, and tags) is absent from the prior same-path submissions;
+    tasks already submitted are skipped. Use this after editing a task file to add only the
+    new work. Cannot be combined with ``--repeat``.
+
 ``-q``, ``--queue``
     Submit directly to live queue instead of database.
 

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/submit_usage.rst

@@ -1,5 +1,5 @@
-``hs`` ``submit`` ``[-h]`` ``[ARGS... | -f FILE]``
+``hs`` ``submit`` ``[-h]`` ``[ARGS... | -f FILE | --from-json SPEC]``
     ``[-q [-H ADDR] [-p NUM] [-k KEY] | --initdb]``
     ``[-b NUM]`` ``[-w SEC]`` ``[-c NUM]`` ``[-m MEM]`` ``[-W SEC]`` ``[-g NUM]``
-    ``[--template CMD]`` ``[-t TAG...]``
+    ``[--repeat | --update]`` ``[--template CMD]`` ``[-t TAG...]``
     ``[--no-tls | [--tls-ca PATH] [--tls-cert PATH] [--tls-key PATH]]``

hypershell-2.9.0a1/docs/_include/task_search_desc.rst

@@ -0,0 +1,18 @@
+Search tasks in the database.
+
+A database must be configured.
+Specifying *FIELD* names defines what is included in the output
+(by default all fields are included).
+
+As a safeguard against accidentally dumping an entire database, ``hs list`` refuses to return
+more than 1,000 tasks unless the intent is made explicit with ``--all`` (dump everything),
+``--limit`` *N* (bound the result), or ``--count`` (count only) - much as ``hs update`` confirms
+a bulk change before applying it. A bare ``hs list`` with no arguments prints the usage statement.
+
+This command maps directly to underlying SQL queries.
+
+When printing to a terminal, task records are colorized by `exit_status`: successful tasks
+(0) are green, cancelled tasks (-1, including those terminated by ``SIGHUP``) are shown faint,
+tasks that terminated abnormally (killed by a signal, or that never ran - any other negative
+status) are yellow, and tasks that ran and returned a non-zero code are red. Tasks that have
+not yet run are left uncolored.

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/task_search_help.rst

@@ -8,6 +8,14 @@ FIELD
 Options
 ^^^^^^^
 
+``--all``
+    Dump all matching tasks, imposing no limit.
+
+    As a guard against accidentally dumping an entire database, ``hs list`` refuses to return
+    more than 1,000 results unless the intent is made explicit. ``--all`` dumps them all,
+    ``--limit`` *N* bounds the result, and ``--count`` only counts them; the three are mutually
+    exclusive (e.g. ``hs list --all`` or ``hs list --all --csv``).
+
 ``-w``, ``--where`` *COND...*
     List of conditional statements to filter results (e.g., ``-w 'duration >= 600'``).
 
@@ -40,9 +48,20 @@ Options
 ``-R``, ``--remaining``
     Alias for ``-w 'exit_status == null'``.
 
+``-X``, ``--cancelled``
+    Alias for ``-w 'exit_status == -1'`` (tasks that were cancelled).
+
 ``--retries``
     Alias for ``-w 'attempt > 1'`` (tasks that have been retried).
 
+``--signal`` *NAME*
+    Match tasks whose process was terminated by signal *NAME* (e.g. ``TERM``, ``KILL``, ``HUP``).
+
+    A task killed by signal *N* is recorded with ``exit_status = -N`` (the convention used by
+    Python's subprocess module). The ``SIG`` prefix is optional and the name is case-insensitive.
+    A task terminated by ``HUP`` lands on -1 and is therefore treated as cancelled
+    (see ``--cancelled``).
+
 ``-f``, ``--format`` *FORMAT*
     Specify output format (either ``normal``, ``plain``, ``table``, ``csv``, ``json``).
 

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/task_update_help.rst

@@ -15,6 +15,12 @@ Options
     becomes terminal: it will not be scheduled, retried, or reverted on ``--restart``.
     A task cannot be cancelled after it is sent to remote clients.
 
+    A running task can also be cancelled *in flight* on the client by terminating its
+    process with ``SIGHUP`` (e.g. ``kill -HUP``). Because a process killed by signal *N* is
+    recorded with ``exit_status = -N``, a ``SIGHUP`` (signal 1) death lands on -1 - the same
+    cancelled state - so such a task is likewise not retried. See ``hs list --cancelled`` and
+    ``hs list --signal HUP``.
+
 ``--revert``
     Revert specified tasks.
 
@@ -68,9 +74,20 @@ Options
 ``-R``, ``--remaining``
     Alias for ``-w 'exit_status == null'``.
 
+``-X``, ``--cancelled``
+    Alias for ``-w 'exit_status == -1'`` (tasks that were cancelled).
+
 ``--retries``
     Alias for ``-w 'attempt > 1'`` (tasks that have been retried).
 
+``--signal`` *NAME*
+    Match tasks whose process was terminated by signal *NAME* (e.g. ``TERM``, ``KILL``, ``HUP``).
+
+    A task killed by signal *N* is recorded with ``exit_status = -N`` (the convention used by
+    Python's subprocess module). The ``SIG`` prefix is optional and the name is case-insensitive.
+    A task terminated by ``HUP`` lands on -1 and is therefore treated as cancelled
+    (see ``--cancelled``).
+
 ``-f``, ``--no-confirm``
     Do not ask for confirmation.
 

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/templates.rst

@@ -15,6 +15,61 @@ These are expanded `prior` to scheduling as the actual `args` for the task.
 
 -------------------
 
+Named Fields
+^^^^^^^^^^^^
+
+|
+
+When tasks are submitted from a JSON file with ``--from-json`` (see the ``submit``
+command), each task object's keys become available as named template fields. A
+``{key}`` pattern expands to that key's value for the given task, and the keys also
+become task tags.
+
+For a task object ``{"seqid": "Chr1", "start": 1, "end": 5000000}`` the template
+
+``process --region {seqid}:{start}-{end}``
+
+expands to
+
+``process --region Chr1:1-5000000``.
+
+Values are stringified: booleans render as ``true``/``false``, ``null`` as an empty
+string, and nested objects/arrays as compact JSON. A built-in pattern (below) takes
+precedence over a task key of the same name. Named fields are resolved at submit time,
+and every ``{key}`` must be present in each task object or the submission is rejected.
+
+-------------------
+
+Execution Slot
+^^^^^^^^^^^^^^
+
+|
+
+Each client runs ``N`` task executors (set with ``-N``/``--num-threads``); every executor owns
+a fixed **slot** for its lifetime, a zero-based index in ``0 .. N-1``. Two patterns expose it so a
+task can claim a non-overlapping share of the node's cores, memory, or GPUs without colliding with
+its siblings.
+
+``{slot}``
+    The executor's zero-based slot index (``0 .. N-1``).
+
+``{slot_count}``
+    The number of executors on this client (``N``).
+
+``taskset -c {slot} ./sim {}``
+    Pin each task to a distinct CPU core.
+
+``CUDA_VISIBLE_DEVICES={slot} python train.py {}``
+    Assign a distinct GPU per task.
+
+These fields are resolved at execution time, when the ``client`` expands the template — the normal
+line-input path, whether database-backed or ``--no-db``. They are *not* available when the template
+is instead expanded at submit time (JSON mode via ``--from-json``, or the ``submit`` command); use
+the equivalent ``TASK_SLOT`` and ``TASK_SLOT_COUNT`` environment variables there, which are always
+defined.
+
+-------------------
+
 Filepath Operations
 ^^^^^^^^^^^^^^^^^^^
 

{hypershell-2.8.2 → hypershell-2.9.0a1}/docs/_include/templates_alt.rst

@@ -13,6 +13,49 @@ argument to all clients.
 In some situations it may be useful to expand a template with the ``submit`` command.
 These are expanded `prior` to scheduling as the actual `args` for the task.
 
+Named Fields
+^^^^^^^^^^^^
+
+When tasks are submitted from a JSON file with ``--from-json`` (see the ``submit``
+command), each task object's keys become available as named template fields. A
+``{key}`` pattern expands to that key's value for the given task, and the keys also
+become task tags.
+
+For a task object ``{"seqid": "Chr1", "start": 1, "end": 5000000}`` the template
+``process --region {seqid}:{start}-{end}`` expands to
+``process --region Chr1:1-5000000``.
+
+Values are stringified: booleans render as ``true``/``false``, ``null`` as an empty
+string, and nested objects/arrays as compact JSON. A built-in pattern (below) takes
+precedence over a task key of the same name. Named fields are resolved at submit time,
+and every ``{key}`` must be present in each task object or the submission is rejected.
+
+Execution Slot
+^^^^^^^^^^^^^^
+
+Each client runs ``N`` task executors (set with ``-N``/``--num-threads``); every executor owns
+a fixed **slot** for its lifetime, a zero-based index in ``0 .. N-1``. Two patterns expose it so a
+task can claim a non-overlapping share of the node's cores, memory, or GPUs without colliding with
+its siblings.
+
+``{slot}``
+    The executor's zero-based slot index (``0 .. N-1``).
+
+``{slot_count}``
+    The number of executors on this client (``N``).
+
+``taskset -c {slot} ./sim {}``
+    Pin each task to a distinct CPU core.
+
+``CUDA_VISIBLE_DEVICES={slot} python train.py {}``
+    Assign a distinct GPU per task.
+
+These fields are resolved at execution time, when the ``client`` expands the template — the normal
+line-input path, whether database-backed or ``--no-db``. They are *not* available when the template
+is instead expanded at submit time (JSON mode via ``--from-json``, or the ``submit`` command); use
+the equivalent ``TASK_SLOT`` and ``TASK_SLOT_COUNT`` environment variables there, which are always
+defined.
+
 Filepath Operations
 ^^^^^^^^^^^^^^^^^^^