flux-batch 0.0.0__tar.gz

flux_batch-0.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022-2023 LLNS, LLC and other HPCIC DevTools Developers.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flux_batch-0.0.0/MANIFEST.in

@@ -0,0 +1,6 @@
+recursive-include flux_batch *
+prune env*
+global-exclude .env
+global-exclude *.py[co]
+recursive-exclude .git *
+global-exclude __pycache__

flux_batch-0.0.0/NOTICE

@@ -0,0 +1,21 @@
+This work was produced under the auspices of the U.S. Department of
+Energy by Lawrence Livermore National Laboratory under Contract
+DE-AC52-07NA27344.
+
+This work was prepared as an account of work sponsored by an agency of
+the United States Government. Neither the United States Government nor
+Lawrence Livermore National Security, LLC, nor any of their employees
+makes any warranty, expressed or implied, or assumes any legal liability
+or responsibility for the accuracy, completeness, or usefulness of any
+information, apparatus, product, or process disclosed, or represents that
+its use would not infringe privately owned rights.
+
+Reference herein to any specific commercial product, process, or service
+by trade name, trademark, manufacturer, or otherwise does not necessarily
+constitute or imply its endorsement, recommendation, or favoring by the
+United States Government or Lawrence Livermore National Security, LLC.
+
+The views and opinions of authors expressed herein do not necessarily
+state or reflect those of the United States Government or Lawrence
+Livermore National Security, LLC, and shall not be used for advertising
+or product endorsement purposes.

flux_batch-0.0.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.1
+Name: flux-batch
+Version: 0.0.0
+Summary: Python SDK for flux batch jobs and services
+Home-page: https://github.com/converged-computing/flux-batch
+Author: Vanessa Sochat
+Author-email: vsoch@users.noreply.github.com
+Maintainer: Vanessa Sochat
+License: LICENSE
+Keywords: flux,flux framework,hpc,batch,workloads
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: C
+Classifier: Programming Language :: Python
+Classifier: Topic :: Software Development
+Classifier: Topic :: Scientific/Engineering
+Classifier: Operating System :: Unix
+Classifier: Programming Language :: Python :: 3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: pyyaml
+Requires-Dist: ply
+Provides-Extra: all
+Requires-Dist: pyyaml; extra == "all"
+Requires-Dist: ply; extra == "all"
+Requires-Dist: pytest>=4.6.2; extra == "all"
+
+# flux-batch
+
+> Python SDK to generate Flux batch jobs and services
+
+[![PyPI version](https://badge.fury.io/py/flux-batch.svg)](https://badge.fury.io/py/flux-batch)
+
+![https://github.com/converged-computing/flux-batch/raw/main/img/flux-batch-small.png](https://github.com/converged-computing/flux-batch/raw/main/img/flux-batch-small.png)
+
+## Related Projects
+
+- [flux-mcp](https://github.com/converged-computing/flux-mcp): MCP functions for Flux.
+- [flux-mcp-server](https://github.com/converged-computing/flux-mcp-server): MCP server.
+- [fractale-mcp](https://github.com/compspec/fractale-mcp): (fractale) MCP orchestration (agents, databases, ui interfaces).
+- [hpc-mcp](https://github.com/converged-computing/hpc-mcp): HPC tools for a larger set of HPC and converged computing use cases.
+
+## Services
+
+ - **flux-scribe**: Write job events to a local sqlite database via the JournalConsumer (not added yet, written and needs testing)
+
+## Usage
+
+This is a small Flux utility that makes it easy to create Flux batch jobs and services.
+The use case is to submit work (one or more jobs) under an instance, and run a custom service, or epilog and prolog commands. We will provision several services here, and you can also provide your own name to start / stop.
+
+### Setup
+
+Install the library and start (or be in) a flux instance.
+
+```bash
+flux start
+pip install -e . --break-system-packages
+```
+
+### Example
+
+Run the controlled example to see a batch job with prolog and epilog run and complete:
+
+```bash
+python3 ./tests/test_flux_batch.py
+```
+```console
+Flux Batch Module Test
+[OK] Connected to Flux.
+[*] Creating batch jobs...
+[*] Mapping attributes to BatchJobspecV1...
+[*] Previewing submission (Dryrun -> Wrapper)...
+#!/bin/bash
+echo 'Batch Wrapper Starting'
+flux submit --wait /bin/echo 'Job 1 starting'
+flux submit --wait /bin/sleep 5
+flux submit --wait /bin/echo 'Job 2 finished'
+flux job wait --all
+echo 'Batch Wrapper Finished'
+[*] Performing submission (Dryrun -> Wrapper -> Submit)...
+[SUCCESS] Batch submitted! Flux Job ID: ƒMX29AwFu
+```
+```bash
+$ flux jobs -a
+       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
+   ƒMX29AwFu vscode   test-batch  R      1      1   4.213s 68e8c4399c15
+```
+```bash
+$ flux jobs -a
+       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
+   ƒMX29AwFu vscode   test-batch CD      1      1   6.354s 68e8c4399c15
+```
+
+Here is an explicit (manual) example to do the same:
+
+```python
+import flux
+import flux_batch
+
+# for pretty printing
+# from rich import print
+
+handle = flux.Flux()
+
+# Create your batch job with some number of commands
+batch = flux_batch.BatchJobV1()
+batch.add_job(["echo", "Job 1 starting"])
+batch.add_job(["sleep", "5"])
+batch.add_job(["echo", "Job 2 finished"])
+
+# Wrap it up into a jobspec
+jobspec = flux_batch.BatchJobspecV1.from_jobs(
+    batch,
+    nodes=1,
+    nslots=1,
+    time_limit="10m",
+    job_name="test-batch"
+)
+
+# Add a prolog and epilog
+jobspec.add_prolog("echo 'Batch Wrapper Starting'")
+jobspec.add_epilog("echo 'Batch Wrapper Finished'")
+
+# Add a service (this assumes user level that exists)
+# jobspec.add_service("my-service'")
+
+# Preview it
+print(flux_batch.submit(handle, jobspec, dry_run=True))
+
+# Submit that bad boi.
+jobid = flux_batch.submit(handle, jobspec)
+```
+
+## TODO
+
+- Option for controlled output (that we can easily get after)
+- Create flux-scribe service, add example (need to test on cluster with systemctl)
+
+## License
+
+HPCIC DevTools is distributed under the terms of the MIT license.
+All new contributions must be made under this license.
+
+See [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),
+[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and
+[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.
+
+SPDX-License-Identifier: (MIT)
+
+LLNL-CODE- 842614

flux_batch-0.0.0/README.md

@@ -0,0 +1,124 @@
+# flux-batch
+
+> Python SDK to generate Flux batch jobs and services
+
+[![PyPI version](https://badge.fury.io/py/flux-batch.svg)](https://badge.fury.io/py/flux-batch)
+
+![https://github.com/converged-computing/flux-batch/raw/main/img/flux-batch-small.png](https://github.com/converged-computing/flux-batch/raw/main/img/flux-batch-small.png)
+
+## Related Projects
+
+- [flux-mcp](https://github.com/converged-computing/flux-mcp): MCP functions for Flux.
+- [flux-mcp-server](https://github.com/converged-computing/flux-mcp-server): MCP server.
+- [fractale-mcp](https://github.com/compspec/fractale-mcp): (fractale) MCP orchestration (agents, databases, ui interfaces).
+- [hpc-mcp](https://github.com/converged-computing/hpc-mcp): HPC tools for a larger set of HPC and converged computing use cases.
+
+## Services
+
+ - **flux-scribe**: Write job events to a local sqlite database via the JournalConsumer (not added yet, written and needs testing)
+
+## Usage
+
+This is a small Flux utility that makes it easy to create Flux batch jobs and services.
+The use case is to submit work (one or more jobs) under an instance, and run a custom service, or epilog and prolog commands. We will provision several services here, and you can also provide your own name to start / stop.
+
+### Setup
+
+Install the library and start (or be in) a flux instance.
+
+```bash
+flux start
+pip install -e . --break-system-packages
+```
+
+### Example
+
+Run the controlled example to see a batch job with prolog and epilog run and complete:
+
+```bash
+python3 ./tests/test_flux_batch.py
+```
+```console
+Flux Batch Module Test
+[OK] Connected to Flux.
+[*] Creating batch jobs...
+[*] Mapping attributes to BatchJobspecV1...
+[*] Previewing submission (Dryrun -> Wrapper)...
+#!/bin/bash
+echo 'Batch Wrapper Starting'
+flux submit --wait /bin/echo 'Job 1 starting'
+flux submit --wait /bin/sleep 5
+flux submit --wait /bin/echo 'Job 2 finished'
+flux job wait --all
+echo 'Batch Wrapper Finished'
+[*] Performing submission (Dryrun -> Wrapper -> Submit)...
+[SUCCESS] Batch submitted! Flux Job ID: ƒMX29AwFu
+```
+```bash
+$ flux jobs -a
+       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
+   ƒMX29AwFu vscode   test-batch  R      1      1   4.213s 68e8c4399c15
+```
+```bash
+$ flux jobs -a
+       JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
+   ƒMX29AwFu vscode   test-batch CD      1      1   6.354s 68e8c4399c15
+```
+
+Here is an explicit (manual) example to do the same:
+
+```python
+import flux
+import flux_batch
+
+# for pretty printing
+# from rich import print
+
+handle = flux.Flux()
+
+# Create your batch job with some number of commands
+batch = flux_batch.BatchJobV1()
+batch.add_job(["echo", "Job 1 starting"])
+batch.add_job(["sleep", "5"])
+batch.add_job(["echo", "Job 2 finished"])
+
+# Wrap it up into a jobspec
+jobspec = flux_batch.BatchJobspecV1.from_jobs(
+    batch,
+    nodes=1,
+    nslots=1,
+    time_limit="10m",
+    job_name="test-batch"
+)
+
+# Add a prolog and epilog
+jobspec.add_prolog("echo 'Batch Wrapper Starting'")
+jobspec.add_epilog("echo 'Batch Wrapper Finished'")
+
+# Add a service (this assumes user level that exists)
+# jobspec.add_service("my-service'")
+
+# Preview it
+print(flux_batch.submit(handle, jobspec, dry_run=True))
+
+# Submit that bad boi.
+jobid = flux_batch.submit(handle, jobspec)
+```
+
+## TODO
+
+- Option for controlled output (that we can easily get after)
+- Create flux-scribe service, add example (need to test on cluster with systemctl)
+
+## License
+
+HPCIC DevTools is distributed under the terms of the MIT license.
+All new contributions must be made under this license.
+
+See [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),
+[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and
+[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.
+
+SPDX-License-Identifier: (MIT)
+
+LLNL-CODE- 842614

flux_batch-0.0.0/flux_batch/__init__.py

@@ -0,0 +1,7 @@
+from .jobspec import BatchJobspecV1
+from .models import BatchAttributesV1, BatchJobV1
+from .submit import submit
+
+__all__ = ["BatchJobV1", "BatchAttributesV1", "BatchJobspecV1", "submit"]
+
+from .version import __version__  # noqa

flux_batch-0.0.0/flux_batch/jobspec.py

@@ -0,0 +1,132 @@
+import shlex
+from typing import List
+
+import flux_batch.models as models
+
+
+class BatchJobspecV1:
+    """
+    A BatchJobspecV1 mirrors a JobspecV1. We need to:
+
+    1. Add some number of commands or a script
+    2. Add optional services (start/stop)
+    """
+
+    def __init__(self, attributes: models.BatchAttributesV1 = None):
+        self.attributes = attributes or models.BatchAttributesV1()
+        self.commands: List[str] = []
+        self.prologs: List[str] = []
+        self.epilogs: List[str] = []
+        self.services: List[str] = []
+
+    @classmethod
+    def from_command(cls, command: List[str], **kwargs):
+        inst = cls(models.BatchAttributesV1(**kwargs))
+        inst.commands = [shlex.join(command)]
+        return inst
+
+    @classmethod
+    def from_jobs(cls, batch: models.BatchJobV1, **kwargs):
+        """
+        Generate the batch script from a set of jobs.
+
+        With more than one job, we assume we are waiting.
+        """
+        inst = cls(models.BatchAttributesV1(**kwargs))
+        if len(batch.jobs) > 1:
+            for job_str in batch.jobs:
+                inst.commands.append(f"flux submit --wait {job_str}")
+            # Assume we want to wait for all jobs
+            inst.commands.append("flux job wait --all")
+        else:
+            inst.commands = batch.jobs
+        return inst
+
+    def add_service(self, service: str):
+        self.services.append(service)
+
+    def add_prolog(self, cmd: str):
+        self.prologs.append(cmd)
+
+    def add_epilog(self, cmd: str):
+        self.epilogs.append(cmd)
+
+    def get_cli_flags(self) -> List[str]:
+        """
+        Converts BatchAttributesV1 into a list of strings for subprocess.
+        """
+        flags = []
+        attr = self.attributes
+
+        # Mapping table for simple flags
+        mapping = {
+            "nslots": "-n",
+            "cores_per_slot": "-c",
+            "gpus_per_slot": "-g",
+            "nodes": "-N",
+            "bank": "-B",
+            "queue": "-q",
+            "time_limit": "-t",
+            "urgency": "--urgency",
+            "job_name": "--job-name",
+            "cwd": "--cwd",
+            "dependency": "--dependency",
+            "requires": "--requires",
+            "begin_time": "--begin-time",
+            "signal": "--signal",
+            "broker_opts": "--broker-opts",
+            "dump": "--dump",
+            "flags": "--flags",
+        }
+
+        for field_name, flag in mapping.items():
+            val = getattr(attr, field_name)
+            if val is not None:
+                flags.extend([flag, str(val)])
+
+        # Boolean flags
+        if attr.exclusive:
+            flags.append("-x")
+        if attr.unbuffered:
+            flags.append("-u")
+        if attr.wrap:
+            flags.append("--wrap")
+
+        # Multi-use flags
+        multi_mapping = {
+            "setopt": "-o",
+            "setattr": "-S",
+            "add_file": "--add-file",
+            "env": "--env",
+            "env_remove": "--env-remove",
+            "env_file": "--env-file",
+            "rlimit": "--rlimit",
+            "conf": "--conf",
+        }
+        for field_name, flag in multi_mapping.items():
+            for val in getattr(attr, field_name):
+                flags.extend([flag, str(val)])
+
+        return flags
+
+    def generate_wrapper_script(self) -> str:
+        """
+        Generate the wrapper script.
+
+        1. Start with hashbang!
+        2. Add prologs
+        3. Add services start
+        4. Add jobs/commands
+        5. Stop services
+        6. And epilogs
+        """
+
+        lines = ["#!/bin/bash"]
+        lines.extend(self.prologs)
+        for s in self.services:
+            lines.append(f"systemctl --user start {s}")
+        lines.extend(self.commands)
+        for s in reversed(self.services):
+            lines.append(f"systemctl --user stop {s}")
+        lines.extend(self.epilogs)
+        return "\n".join(lines)

flux_batch-0.0.0/flux_batch/logger/__init__.py

@@ -0,0 +1 @@
+from .logger import LogColors, logger, setup_logger

flux_batch-0.0.0/flux_batch/logger/generate.py

@@ -0,0 +1,208 @@
+from random import choice
+
+
+class JobNamer:
+    _descriptors = [
+        "chunky",
+        "buttery",
+        "delicious",
+        "scruptious",
+        "dinosaur",
+        "boopy",
+        "lovely",
+        "carnivorous",
+        "hanky",
+        "loopy",
+        "doopy",
+        "astute",
+        "gloopy",
+        "outstanding",
+        "stinky",
+        "conspicuous",
+        "fugly",
+        "frigid",
+        "angry",
+        "adorable",
+        "sticky",
+        "moolicious",
+        "cowy",
+        "spicy",
+        "grated",
+        "crusty",
+        "stanky",
+        "blank",
+        "bumfuzzled",
+        "fuzzy",
+        "hairy",
+        "peachy",
+        "tart",
+        "creamy",
+        "arid",
+        "strawberry",
+        "butterscotch",
+        "wobbly",
+        "persnickety",
+        "nerdy",
+        "dirty",
+        "placid",
+        "bloated",
+        "swampy",
+        "pusheena",
+        "hello",
+        "goodbye",
+        "milky",
+        "purple",
+        "rainbow",
+        "bricky",
+        "muffled",
+        "anxious",
+        "misunderstood",
+        "eccentric",
+        "quirky",
+        "lovable",
+        "reclusive",
+        "faux",
+        "evasive",
+        "confused",
+        "crunchy",
+        "expensive",
+        "ornery",
+        "fat",
+        "phat",
+        "joyous",
+        "expressive",
+        "psycho",
+        "chocolate",
+        "salted",
+        "gassy",
+        "red",
+        "blue",
+    ]
+
+    _nouns = [
+        "squidward",
+        "hippo",
+        "butter",
+        "animal",
+        "peas",
+        "lettuce",
+        "carrot",
+        "onion",
+        "peanut",
+        "cupcake",
+        "muffin",
+        "buttface",
+        "leopard",
+        "parrot",
+        "parsnip",
+        "poodle",
+        "itch",
+        "punk",
+        "kerfuffle",
+        "soup",
+        "noodle",
+        "avocado",
+        "peanut-butter",
+        "latke",
+        "milkshake",
+        "banana",
+        "lizard",
+        "lemur",
+        "lentil",
+        "bits",
+        "house",
+        "leader",
+        "toaster",
+        "signal",
+        "pancake",
+        "kitty",
+        "cat",
+        "cattywampus",
+        "poo",
+        "malarkey",
+        "general",
+        "rabbit",
+        "chair",
+        "staircase",
+        "underoos",
+        "snack",
+        "lamp",
+        "eagle",
+        "hobbit",
+        "diablo",
+        "earthworm",
+        "pot",
+        "plant",
+        "leg",
+        "arm",
+        "bike",
+        "citrus",
+        "dog",
+        "puppy",
+        "blackbean",
+        "ricecake",
+        "gato",
+        "nalgas",
+        "lemon",
+        "caramel",
+        "fudge",
+        "cherry",
+        "sundae",
+        "truffle",
+        "cinnamonbun",
+        "pastry",
+        "egg",
+        "omelette",
+        "fork",
+        "knife",
+        "spoon",
+        "salad",
+        "train",
+        "car",
+        "motorcycle",
+        "bicycle",
+        "platanos",
+        "mango",
+        "taco",
+        "pedo",
+        "nunchucks",
+        "destiny",
+        "hope",
+        "despacito",
+        "frito",
+        "chip",
+    ]
+
+    def generate(self, delim="-", length=4, chars="0123456789"):
+        """
+        Generate a robot name. Inspiration from Haikunator, but much more
+                 poorly implemented ;)
+
+        Parameters
+        ==========
+        delim: Delimiter
+        length: TokenLength
+        chars: TokenChars
+        """
+
+        descriptor = self._select(self._descriptors)
+        noun = self._select(self._nouns)
+        numbers = "".join((self._select(chars) for _ in range(length)))
+        return delim.join([descriptor, noun, numbers])
+
+    def _select(self, select_from):
+        """select an element from a list using random.choice
+
+        Parameters
+        ==========
+        should be a list of things to select from
+        """
+        if not select_from:
+            return ""
+
+        return choice(select_from)
+
+
+def generate_name():
+    namer = JobNamer()
+    return namer.generate()