oh-my-batch 0.2.0__tar.gz → 0.2.2__tar.gz

oh_my_batch-0.2.0/README.md → oh_my_batch-0.2.2/PKG-INFO

@@ -1,22 +1,57 @@
+Metadata-Version: 2.1
+Name: oh-my-batch
+Version: 0.2.2
+Summary: 
+License: GPL
+Author: weihong.xu
+Author-email: xuweihong.cn@gmail.com
+Requires-Python: >=3.8,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: fire (>=0.7.0,<0.8.0)
+Description-Content-Type: text/markdown
+
 # oh-my-batch
 
 [![PyPI version](https://badge.fury.io/py/oh-my-batch.svg)](https://badge.fury.io/py/oh-my-batch)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/oh-my-batch)](https://pypi.org/project/oh-my-batch/)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/oh-my-batch)](https://pypi.org/project/oh-my-batch/)
 
-A simple tool to manipulate batch tasks designed for scientific computing community.
+A toolkit to manipulate batch tasks with command line. Designed for scientific computing community.
 
 ## Features
 * `omb combo`: generate folders/files from different combinations of parameters
 * `omb batch`: generate batch scripts from multiple working directories
 * `omb job`: track the state of job in job schedular
+* `omb misc`: miscellaneous commands
 
 ## Install
 ```bash
 pip install oh-my-batch
 ```
 
+## Shell tips
+`oh-my-batch` is intended to help you implement computational workflows with shell scripts.
+To make the best use of `oh-my-batch`, you need to know some shell tips.
+
+* [Retry commands until success in shell script](https://stackoverflow.com/a/79191004/3099733)
+* [Run multiple line shell script with ssh](https://stackoverflow.com/a/32082912/3099733)
+
 ## Use cases
+### Load functions in shell script
+You can load useful functions from `oh-my-batch` this way:
+
+```bash
+omb misc export-shell-func > omb-func.sh && source omb-func.sh && rm omb-func.sh
+```
+
+This will load some functions to your shell script, for example, `checkpoint`.
 
 ### Generate files from different combinations of parameters
 
@@ -34,8 +69,8 @@ touch tmp/1.data tmp/2.data tmp/3.data
 
 # prepare a lammps input file template
 cat > tmp/in.lmp.tmp <<EOF
-read_data $DATA_FILE
-velocity all create $TEMP $RANDOM
+read_data @DATA_FILE
+velocity all create @TEMP @RANDOM
 run 1000
 EOF
 
@@ -109,3 +144,4 @@ and wait for the jobs to finish. If the job fails, it will retry for at most 3 t
 The `--recovery` option will save the job information to `lammps-jobs.json` file.
 If `omb job` is interrupted, you can rerun the exact same command to recover the job status,
 so that you don't need to resubmit the jobs that are still running or completed.
+

oh_my_batch-0.2.0/PKG-INFO → oh_my_batch-0.2.2/README.md

@@ -1,40 +1,38 @@
-Metadata-Version: 2.1
-Name: oh-my-batch
-Version: 0.2.0
-Summary: 
-License: GPL
-Author: weihong.xu
-Author-email: xuweihong.cn@gmail.com
-Requires-Python: >=3.8,<4.0
-Classifier: License :: Other/Proprietary License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: fire (>=0.7.0,<0.8.0)
-Description-Content-Type: text/markdown
-
 # oh-my-batch
 
 [![PyPI version](https://badge.fury.io/py/oh-my-batch.svg)](https://badge.fury.io/py/oh-my-batch)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/oh-my-batch)](https://pypi.org/project/oh-my-batch/)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/oh-my-batch)](https://pypi.org/project/oh-my-batch/)
 
-A simple tool to manipulate batch tasks designed for scientific computing community.
+A toolkit to manipulate batch tasks with command line. Designed for scientific computing community.
 
 ## Features
 * `omb combo`: generate folders/files from different combinations of parameters
 * `omb batch`: generate batch scripts from multiple working directories
 * `omb job`: track the state of job in job schedular
+* `omb misc`: miscellaneous commands
 
 ## Install
 ```bash
 pip install oh-my-batch
 ```
 
+## Shell tips
+`oh-my-batch` is intended to help you implement computational workflows with shell scripts.
+To make the best use of `oh-my-batch`, you need to know some shell tips.
+
+* [Retry commands until success in shell script](https://stackoverflow.com/a/79191004/3099733)
+* [Run multiple line shell script with ssh](https://stackoverflow.com/a/32082912/3099733)
+
 ## Use cases
+### Load functions in shell script
+You can load useful functions from `oh-my-batch` this way:
+
+```bash
+omb misc export-shell-func > omb-func.sh && source omb-func.sh && rm omb-func.sh
+```
+
+This will load some functions to your shell script, for example, `checkpoint`.
 
 ### Generate files from different combinations of parameters
 
@@ -52,8 +50,8 @@ touch tmp/1.data tmp/2.data tmp/3.data
 
 # prepare a lammps input file template
 cat > tmp/in.lmp.tmp <<EOF
-read_data $DATA_FILE
-velocity all create $TEMP $RANDOM
+read_data @DATA_FILE
+velocity all create @TEMP @RANDOM
 run 1000
 EOF
 
@@ -127,4 +125,3 @@ and wait for the jobs to finish. If the job fails, it will retry for at most 3 t
 The `--recovery` option will save the job information to `lammps-jobs.json` file.
 If `omb job` is interrupted, you can rerun the exact same command to recover the job status,
 so that you don't need to resubmit the jobs that are still running or completed.
-

{oh_my_batch-0.2.0 → oh_my_batch-0.2.2}/oh_my_batch/assets/functions.sh

@@ -1,6 +1,7 @@
+GREEN='\033[0;32m'
+NC='\033[0m'
 
 checkpoint() {
-    # Usage: checkpoint <flag_file> <command> [arg1] [arg2] ...
     local flag_file="$1"
     shift  # Remove the first argument so $@ contains only the command and its arguments
     if [ -f "$flag_file" ]; then
@@ -10,6 +11,7 @@ checkpoint() {
         local exit_code=$?
         if [ $exit_code -eq 0 ]; then
             local current_time=$(date '+%Y-%m-%d %H:%M:%S')
+            mkdir -p "$(dirname "$flag_file")"
             printf 'Command succeeded at %s\n' "$current_time" > "$flag_file"
             echo "Created flag file '$flag_file' with timestamp: $current_time"
         else
@@ -18,3 +20,16 @@ checkpoint() {
         fi
     fi
 }
+
+echo -e "${GREEN}Functiion: checkpoint${NC}"
+cat <<EOF
+Usage:
+    checkpoint <flag_file> <command> [arg1] [arg2] ...
+
+Set a flag file to indicate the completion of a command,
+so that it will not be executed again next time.
+
+Example:
+    checkpoint lmp.done lmp -in in.lmp
+
+EOF

{oh_my_batch-0.2.0 → oh_my_batch-0.2.2}/oh_my_batch/cli.py

@@ -23,6 +23,9 @@ class OhMyBatch:
     def job(self):
         return JobCli()
 
+    def misc(self):
+        from .misc import Misc
+        return Misc()
 
 def main():
     fire.Fire(OhMyBatch)

{oh_my_batch-0.2.0 → oh_my_batch-0.2.2}/oh_my_batch/combo.py

@@ -1,6 +1,7 @@
 from itertools import product
 from string import Template
 import random
+import json
 import os
 
 from .util import expand_globs, mode_translate
@@ -67,7 +68,7 @@ class ComboMaker:
         self.add_var(key, *args, broadcast=broadcast)
         return self
 
-    def add_files(self, key: str, *path: str, broadcast=False, abs=False):
+    def add_files(self, key: str, *path: str, broadcast=False, abs=False, raise_invalid=False):
         """
         Add a variable with files by glob pattern
         For example, suppose there are 3 files named 1.txt, 2.txt, 3.txt in data directory,
@@ -78,8 +79,9 @@ class ComboMaker:
         :param path: Path to files, can include glob pattern
         :param broadcast: If True, values are broadcasted, otherwise they are producted when making combos
         :param abs: If True, path will be turned into absolute path
+        :param raise_invalid: If True, will raise error if no file found for a glob pattern
         """
-        args = expand_globs(path, raise_invalid=True)
+        args = expand_globs(path, raise_invalid=raise_invalid)
         if not args:
             raise ValueError(f"No files found for {path}")
         if abs:
@@ -87,7 +89,8 @@ class ComboMaker:
         self.add_var(key, *args, broadcast=broadcast)
         return self
 
-    def add_files_as_one(self, key: str, path: str, broadcast=False, sep=' ', abs=False):
+    def add_files_as_one(self, key: str, *path: str, broadcast=False, format=None,
+                         sep=' ', abs=False, raise_invalid=False):
         """
         Add a variable with files by glob pattern as one string
         Unlike add_files, this function joins the files with a delimiter.
@@ -98,15 +101,25 @@ class ComboMaker:
         :param key: Variable name
         :param path: Path to files, can include glob pattern
         :param broadcast: If True, values are broadcasted, otherwise they are producted when making combos
+        :param format: the way to format the files, can be None, 'json-list','json-item'
         :param sep: Separator to join files
         :param abs: If True, path will be turned into absolute path
+        :param raise_invalid: If True, will raise error if no file found for a glob pattern
         """
-        args = expand_globs(path, raise_invalid=True)
+        args = expand_globs(path, raise_invalid=raise_invalid)
         if not args:
             raise ValueError(f"No files found for {path}")
         if abs:
             args = [os.path.abspath(p) for p in args]
-        self.add_var(key, sep.join(args), broadcast=broadcast)
+        if format is None:
+            value = sep.join(args)
+        elif format == 'json-list':
+            value = json.dumps(args)
+        elif format == 'json-item':
+            value = json.dumps(args).strip('[]')
+        else:
+            raise ValueError(f"Invalid format: {format}")
+        self.add_var(key, value, broadcast=broadcast)
         return self
 
     def add_var(self, key: str, *args, broadcast=False):
@@ -148,19 +161,20 @@ class ComboMaker:
                 raise ValueError(f"Variable {key} not found")
         return self
 
-    def make_files(self, template: str, dest: str, delimiter='$', mode=None):
+    def make_files(self, template: str, dest: str, delimiter='@', mode=None):
         """
         Make files from template
         The template file can include variables with delimiter.
-        For example, if delimiter is '$', then the template file can include $var1, $var2, ...
+        For example, if delimiter is '@', then the template file can include @var1, @var2, ...
 
         The destination can also include variables in string format style.
-        For example, if dest is 'output/{i}.txt', then files are saved as output/0.txt, output/1.txt, ...
+        For example, if dest is 'output/{i}-{TEMP}.txt', 
+        then files are saved as output/0-300K.txt, output/1-400K.txt, ...
 
         :param template: Path to template file
         :param dest: Path pattern to destination file
-        :param delimiter: Delimiter for variables in template, default is '$',
-        can be changed to other character, e.g $$, @, ...
+        :param delimiter: Delimiter for variables in template, default is '@', as '$' is popular in shell scripts
+        can be changed to other character, e.g $, $$, ...
         """
         _delimiter = delimiter
 

oh_my_batch-0.2.2/oh_my_batch/misc.py

@@ -0,0 +1,14 @@
+from .assets import get_asset
+
+class Misc:
+
+    def export_shell_func(self):
+        """
+        Export shell functions for batch scripts
+
+        For example, you can load them to you shell environment with the following command:
+            omb misc export-shell-func > omb-func.sh && source omb-func.sh
+        """
+        shell_func = get_asset('functions.sh')
+        with open(shell_func, 'r', encoding='utf-8') as f:
+            print(f.read())

{oh_my_batch-0.2.0 → oh_my_batch-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "oh-my-batch"
-version = "0.2.0"
+version = "0.2.2"
 description = ""
 authors = ["weihong.xu <xuweihong.cn@gmail.com>"]
 license = "GPL"
@@ -11,7 +11,6 @@ packages = [{include = "oh_my_batch"}]
 python = "^3.8"
 fire = "^0.7.0"
 
-
 [build-system]
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"