goPEST 0.0.11__py3-none-any.whl

goPEST-0.0.11.dist-info/LICENSE

@@ -0,0 +1,11 @@
+Copyright 2022 Angus Yeh, University of Auckland, New Zealand
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

goPEST-0.0.11.dist-info/METADATA

@@ -0,0 +1,95 @@
+Metadata-Version: 2.1
+Name: goPEST
+Version: 0.0.11
+Summary: A TOUGH2/Waiwera interface to PEST
+Author-email: Angus Yeh <a.yeh@auckland.ac.nz>
+License: BSD 3-Clause License
+Keywords: goPEST,PEST,Waiwera,TOUGH2,AUTOUGH2
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Scientific/Engineering
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=2.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyTOUGH
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: h5py
+Requires-Dist: pyyaml
+Requires-Dist: tomlkit
+Requires-Dist: xlwt
+Requires-Dist: setuptools-scm
+
+# **goPEST**
+
+goPEST is a set of utilities used to interface PEST with Waiwera and (AU)TOUGH2 simulators.
+
+
+## Install
+
+The easiest way to install goPEST is:
+
+```python -m pip install gopest```
+
+
+## Input files
+
+User needs to prepare a few files for goPEST to work:
+
+- `goPESTconfig.toml`, this file contains all settings/configurations related to the running of goPEST.  It is written in TOML file format.  The file should be placed in the project folder, where user runs gopest commands from.  It is possible to let goPEST generate one with default values.  The file tries to be self-explanatory with comments.
+
+- `goPESTpar.list` is where user specifies model parameters for PEST
+
+- `goPESTobs.list` is where user specifies model observations for PEST
+
+- `goPESTuser.py` is optional if user wish do perform customised setups when the simulation transit from one stage to the next stage in sequence of simulator runs (eg. usually `ns` and `pr` normally natural state then followed by production history).
+
+## Basic usage
+
+The main CLI script `gopest` is to be followed by COMMAND and associated arguments:
+
+```gopest COMMAND [ARGUMENTS]```
+
+To get a list of supported COMMANDs, type `gopest help`.
+
+The first step is to initialise the working directory:
+
+```gopest init```
+
+will setup the current folder.  If all goes well, user can simply run the command:
+
+```gopest submit``` or
+```gopest run``` 
+
+to start the PEST run.  Command `submit` is for the NeSI cluster environment using SLURM.  Job(s) will be submitted to the cluster queue.  PEST master and agents will be launched automatically.  On a local machine where user has full access `gopest run` is used.
+
+## How goPEST works
+
+Several tasks were performed by the `init` command:
+
+- copy user's model files into what goPEST uses internally, these are the `real_model_xxx.*` files, in the current folder, which is also the master folder where PEST is expected to work on.
+
+- go through `goPESTpar.list`, extract and set up parameter data in the PEST control file (usually `case.pst`).  The corresponding `.tpl` files etc required by PEST will be set up.  Note the parameters used in the real model will be extracted and used as the initial parameters in the PEST.
+
+- go through `goPESTobs.list`, and set up observation data in the PEST control file.  Corresponding PEST instruction file `.ins` will also be set up automatically.
+
+## Development notes
+
+- install an editable version of goPEST:
+
+```python -m pip install -e /path/to/repo/root```
+
+- run tests at the root of the repo:
+
+```python -m pytest```
+
+- generalised model sequence runner? now loads user goPESTuser.py, but internal needs to generalise to have more than two run sequence
+
+- in run_ns_pr, code shouldn't worry about nesi/cluster related things, maybe
+  not running local vs nesi either
+
+- remove obsreref related things, use pest_hp now
+
+- I have checked a few PEST related Python libraries.  I am looking for something small and pure for basic editing of PEST control file.  But these are too big for my liking.  I should reconsider about using them.

goPEST-0.0.11.dist-info/RECORD

@@ -0,0 +1,30 @@
+gopest/__init__.py,sha256=sQIeyIA_BvmrEsXURFJskDg6rlPDNJLjFoFJcHn-MkQ,374
+gopest/_version.py,sha256=TLLlZVpoPCYB-ad4nVSzPokzjT6xk0F2KolI_iEJXtA,429
+gopest/check_slaves.py,sha256=PdpDHRjHHl1fJBwhXb8mAhSt6QXK1cJQ03goUXjyumE,13769
+gopest/commands.py,sha256=xXERkb5hb_hbfUZMy1qdD78q1ZakHH5x-1WSyKxzs14,2651
+gopest/common.py,sha256=BT_DqgpqvqnM8obpyt9IOg5OuyqSN1aWuBKqfrUYyu4,7664
+gopest/make_case_pst.py,sha256=DNb1Ta_HE2uYwzlFti9tZRwF80pyvWyavws77LAbSfM,8760
+gopest/obs.py,sha256=rFnWIddecp4Kwx_OHDIbt5uxHTnYI_C3YcaWoznY8cs,10047
+gopest/obs_def.py,sha256=oFYOD8S7HnA1bSRVLRrlKtqBlpwxbcWu_izf88U6ZZo,78308
+gopest/par.py,sha256=lm21mqbwL58I9gLv67mCrVvMC_6BVK7L_4kblTuvgtc,13043
+gopest/par_def.py,sha256=gT-7BTk7zYTYzc3U8WKIRxXAfxIlykTYpvtJbNKBGxM,10841
+gopest/pest_model.py,sha256=KqL1mSrOqP-T3ljyE8UiLsa7FvlEgohnEvai8HiT-Q4,8666
+gopest/rename_latest_files.py,sha256=gZwbZMwni0ZQPjCgOL3C6JG65jO9gFPR9vN9QftqRFY,1095
+gopest/run_beopest.py,sha256=VeuLYI2Wr2leocV92Rowm_s38B_04eJ1dcV1WiMFCe0,7077
+gopest/run_ns_pr.py,sha256=tC5Umidn1gZBHRm2XoM-7FSUlMwWKa9WRxA0C1yfG0U,22721
+gopest/submit_beopest.py,sha256=Q5Z-o6Lz4uuGN8di7NojtiUlRUNHZI6KY_Mmhs30reE,35142
+gopest/data/case.pst,sha256=xGNftyacW3twbqmPUN8V3ZQuNaMyqoGJB0o7Y327fso,2506
+gopest/data/goPESTconfig.aut2.toml,sha256=mrMtwqKy1_4R6qQAPNqLBcMxqVKzrdmSbPpg6rEzD9g,2782
+gopest/data/goPESTconfig.toml,sha256=GbcWFMzStaIEwgStPVyxE7RaIBMVCpZKbMk98r8YdZU,2767
+gopest/data/goPESTobs.list,sha256=aNdlcoP8MQVvsgHaInCnQ_bktFvtiBFCWiPwWI_lkEs,10351
+gopest/data/goPESTpar.list,sha256=Ne-OEsOVh_JtLBVF8HMG5EOmYl8g_SD5mWsbExD2oq4,1852
+gopest/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+gopest/utils/gener_groups.py,sha256=VtbX35VWKE0V0wVBJnUXC0KtcfY-fIz4XrV3OBSYngI,6624
+gopest/utils/t2listingh5.py,sha256=36X-XUdakSkRSn26ZdU8uoAKEfsFydiT0p3xzsobjGk,16791
+gopest/utils/waiwera_listing.py,sha256=c7moQY8DQEH_MqLUuhrinnkEHnDMKWx4srChffXoK6I,27787
+goPEST-0.0.11.dist-info/LICENSE,sha256=ptsyzynWmtAQdzQN1KIqWAkMbJMyKG03PzYniyCa1TE,1490
+goPEST-0.0.11.dist-info/METADATA,sha256=8OBaPfrqRIAxA0-qAuic1YCoTohqWuJemiR4IZL1Zkc,3885
+goPEST-0.0.11.dist-info/WHEEL,sha256=OVMc5UfuAQiSplgO0_WdW7vXVGAt9Hdd6qtN4HotdyA,91
+goPEST-0.0.11.dist-info/entry_points.txt,sha256=lQZ0lnYLOBrcoQehLV6TZsLFu1OJPLcw-PCOUs0_SpA,54
+goPEST-0.0.11.dist-info/top_level.txt,sha256=g3FO8XhhpDnQXWy5B9lv4uMKMEilq1TKv8SAk5x8_eU,7
+goPEST-0.0.11.dist-info/RECORD,,

goPEST-0.0.11.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (75.2.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

goPEST-0.0.11.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+gopest = gopest.commands:gopest_cli

goPEST-0.0.11.dist-info/top_level.txt

@@ -0,0 +1 @@
+gopest

gopest/__init__.py

@@ -0,0 +1,11 @@
+try:
+    # works if calling from an repo or an pip install -e installation
+    from setuptools_scm import get_version
+    __version__ = get_version()
+except LookupError:
+    try:
+        # works if calling from other pip install methods
+        from ._version import version as __version__
+    except ImportError:
+        # worst case scenario
+        __version__ = '0.0.0'

gopest/_version.py

@@ -0,0 +1,16 @@
+# file generated by setuptools_scm
+# don't change, don't track in version control
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple, Union
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+else:
+    VERSION_TUPLE = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+
+__version__ = version = '0.0.11'
+__version_tuple__ = version_tuple = (0, 0, 11)

gopest/check_slaves.py

@@ -0,0 +1,402 @@
+from multiprocessing import Pool, cpu_count
+import time
+import json
+import os
+import os.path
+import sys
+import shutil
+import glob
+import subprocess
+import collections.abc
+
+import yaml
+import tomlkit
+import xlwt
+
+from gopest.common import config
+from gopest.common import runtime
+
+def nested_dict_update(d, u):
+    """ update a nested dict object with an update dict
+    https://stackoverflow.com/a/3233356/2368167
+    """
+    for k, v in u.items():
+        if isinstance(v, collections.abc.Mapping):
+            d[k] = nested_dict_update(d.get(k, {}), v)
+        else:
+            d[k] = v
+    return d
+
+def unix2dos(fname):
+    import sys
+    with open(fname, 'rb') as infile:
+        instr = infile.read()
+    outstr = instr.replace( b"\r\n", b"\n" ).replace( b"\r", b"\n" ).replace( b"\n", b"\r\n" )
+    if len(outstr) == len(instr):
+        return
+    with open(fname, 'wb') as outfile:
+        outfile.write( outstr )
+
+def fixpcf_noptmax(fpst, noptmax=0):
+    """ update PEST case file (.pst) with NOPTMAX, number of optimisation
+    iterations
+    """
+    fpst_bk = fpst + '.backup'
+    if not os.path.isfile(fpst):
+        print('Error: %s does not exist.' % fpst)
+        exit(1)
+    if os.path.isfile(fpst_bk):
+        os.remove(fpst_bk)
+    os.rename(fpst, fpst_bk)
+
+    try:
+        with open(fpst_bk,'r') as fin, open(fpst, 'w') as fout:
+            linesep = None # use current file's line ending
+            isec = 7 # start with 7, so that it will never be 7 again
+            for line in fin:
+                # detect current file's line ending first
+                if linesep is None:
+                    if line[-2] == '\r' and line[-1] == '\n':
+                        linesep = '\r\n'
+                    elif line[-1] == '\n':
+                        linesep = '\n'
+                # find correct line, replace
+                isec += 1
+                oline = line
+                if line.strip().startswith('*') and 'control data' in line:
+                    isec = 0
+                if isec == 7:
+                    # replace first item in line (NOPTMAX)
+                    sps = line.split()
+                    oline = ' ' + ' '.join([str(noptmax)] + sps[1:] + [linesep])
+                fout.write(oline)
+    except Exception as e:
+        fin.close()
+        fout.close()
+        print('fixpcf_noptmax() failed to proceed, restoring...')
+        os.remove(fpst)
+        os.rename(fpst_bk, fpst)
+        raise(e)
+
+def read_rec(rec_file):
+    values = {}
+    with open(rec_file, 'r') as f:
+        reading = False
+        for line in f:
+            if reading:
+                if empty_line:
+                    empty_line = False
+                    continue
+                if not line.strip():
+                    continue
+                if '----->' in line:
+                    break
+                sps = line.strip().split('=')
+                v = float(sps[-1])
+                s = sps[0].strip()
+                if s == 'Sum of squared weighted residuals (ie phi)':
+                    k = 'phi'
+                elif s == 'Optimised measurement objective function':
+                    k = 'phi measured'
+                elif s == 'Optimised regularisation objective function':
+                    k =  'phi regularisation'
+                elif '"' in s:
+                    k = s.split('"')[1]
+                else:
+                    k, v = None, None
+                if k is not None:
+                    values[k] = v
+            if line.startswith('Objective function ----->'):
+                reading, empty_line = True, True
+    return values
+
+class message(object):
+    """ A Waiwera log (.yaml) message object """
+    def __init__(self, msg):
+        self._msg = msg
+        self.level, self.source, self.event, self.data = msg
+
+
+def check_sim_ends(spath):
+    """ extract summary of waiwera simulation from yaml log """
+    def waiwera_ends(fyaml):
+        """ return elapsed time in seconds of waiwera simulation time, None if not
+        successful """
+        try:
+            with open(fyaml) as f:
+                data = yaml.load(f, Loader=yaml.CSafeLoader)
+                r = {''}
+                if data[-3][:3] == ['info', 'timestep', 'end']:
+                    time = data[-3][3]['time']
+                    size = data[-3][3]['size']
+                if data[-1][:3] == ['info', 'simulation', 'destroy']:
+                    elapsed = data[-1][3]['elapsed_seconds']
+                return {'endtime': time, 'stepsize': size, 'elapsed': elapsed}
+        except:
+            return {}
+    cwd = os.getcwd()
+    os.chdir(spath)
+    print('Working in %s...' % spath)
+
+    results = {'run': {}}
+    sequence = config['model']['sequence']
+    input_typ = config['simulator']['input-type']
+    flsts = runtime['filename']['lst_seq']
+    for i,seq in enumerate(sequence):
+        results['run'][seq] = {}
+        if input_typ== 'waiwera':
+            fyaml = flsts[i].replace('.h5', '.yaml')
+            results['run'][seq] = waiwera_ends(fyaml)
+        else:
+            raise NotImplementedError()
+
+    os.chdir(cwd)
+    # print('Restored directory %s' % cwd)
+    return results
+
+def check_run_status(spath):
+    """ extract summary of waiwera simulation from yaml log """
+    cwd = os.getcwd()
+    os.chdir(spath)
+    print('Working in %s...' % spath)
+
+    results = {'run': {}}
+    sequence = config['model']['sequence']
+    input_typ = config['simulator']['input-type']
+    flsts = runtime['filename']['lst_seq']
+    for i,seq in enumerate(sequence):
+        results['run'][seq] = {}
+        results['run'][seq]['exists'] = os.path.exists(flsts[i])
+
+    os.chdir(cwd)
+    # print('Restored directory %s' % cwd)
+    return results
+
+def get_obj_fn(spath):
+    """ Return PEST calculated objective function values by modifying config and
+    PEST case file to run a dummy run.  Then read results from case .rec file.
+
+    This is more destructive, but I have decided to leave those modification
+    in place after running the command for a couple of reasons:
+    1. (lazy) no need to handle undo the mod if exception is raised
+    2. (safe) it's probably safer to leave as modified here, reduce possibility
+       for user to wipe out existing model output files, which can be costly to
+       run
+
+    Unfortunately I need to modify the config.toml file, because gopest run-
+    pest-model is involked by PEST, not this script directly.
+    """
+    cwd = os.getcwd()
+    os.chdir(spath)
+    print('Working in %s...' % spath)
+
+    # only works if the final real model output file exists
+    flst = runtime['filename']['lst_seq'][-1]
+    if not os.path.exists(flst):
+        print('Real model output file %s is missing, cannot get obj. fn.' % flst)
+        os.chdir(cwd)
+        return {}
+
+    # modify config to skip model run
+    with open('goPESTconfig.toml', 'r') as f:
+        cfg = tomlkit.load(f)
+    cfg['model']['skip'] = True
+    with open('goPESTconfig.toml', 'w') as f:
+        tomlkit.dump(cfg, f)
+
+    # modify NOPTMAX in control data .pst file
+    fpst = config['pest']['case-name'] + '.pst'
+    fixpcf_noptmax(fpst, 0)
+
+    cmd = [
+        os.path.join(config['pest']['dir'], config['pest']['executable']),
+        config['pest']['case-name'],
+        ]
+    print('Running: ', cmd)
+    results = {}
+    subprocess.call(cmd)
+    results['obj-fn'] = read_rec('case_reg.rec')
+
+    os.chdir(cwd)
+    # print('Restored directory %s' % cwd)
+    return results
+
+def export_xls(data):
+    phis, regs, others = [], [], []
+    for sln,sl in data.items():
+        if 'obj-fn' in sl:
+            for n in sorted(sl['obj-fn'].keys()):
+                if n.startswith('phi'):
+                    phis.append(n)
+                elif n.startswith('regul_'):
+                    regs.append(n)
+                else:
+                    others.append(n)
+            phis, regs, others = sorted(phis), sorted(regs), sorted(others)
+            break
+    runcs = []
+    for sln,sl in data.items():
+        if 'run' in sl:
+            for rn,r in sl['run'].items():
+                runcs += [rn+'.'+c for c in sorted(r.keys())]
+            break
+    cols = ['slave'] + phis + others + regs + runcs
+
+    wb = xlwt.Workbook()
+    ws = wb.add_sheet('slaves')
+
+    i = 0
+    for j,c in enumerate(cols):
+        ws.write(i, j, c)
+
+    for sln,sl in data.items():
+        if 'obj-fn' in sl:
+            i += 1
+            j = 0
+            ws.write(i, j, sln)
+            for p in (phis + others + regs):
+                j += 1
+                if p in sl['obj-fn']:
+                    ws.write(i, j, sl['obj-fn'][p])
+            for r in runcs:
+                j += 1
+                rn,sn = r.split('.')
+                if 'run' in sl and rn in sl['run'] and sn in sl['run'][rn]:
+                    ws.write(i, j, sl['run'][rn][sn])
+    wb.save('goPESTslaves.xls')
+
+def init_slave(cfg, rt):
+    """ used in multiprocessing Pool to initialise threads with the the updated
+    gopest.common.config and gopest.common.runtime
+    """
+    print('Updating local thread common.config and common.runtime.')
+    global config, runtime
+    config = cfg
+    runtime = rt
+
+hlp = '''
+Usage: gopest check-slaves [--help] [--status] [--end-time] [--obj-fn]
+                           [--dir path_to_slaves] [--pest-exe pest_executable]
+                           [--export-xls]
+
+The check-slaves command searches through slave directories and obtain/collect
+their running status etc.  By default, the pest.slave_dirs property from
+goPESTconfig.toml is used. It is possible to specify a different directory by
+using argument "--dir".
+
+Extracted information will be dumped into a JSON file "goPESTslaves.json".  Note
+that the command will only update/append results into the JSON file if exists.
+It tries not to destroy whatever that is already in the file.
+
+"--status" is the fastest option, which only checks if model output files exist
+in the lsave directories.  (ie. did the runs produce any output files at all)
+
+"--end-time" enables extraction of simulation end time (from output YAML file if
+running waiwera as simulator).
+
+"--obj-fn" is more destructive.  Within each slave directory, it will run PEST
+(with the NOPTMAX set to 0) once with the internal settings set to skip actual
+model runs.  This essentially runs PEST so that observations and objective
+function will be extracted from model outputs within the slave directory.
+
+"--pest-exe" is useful with the "--obj-fn" option when user have the slaves
+directories in a different environment than where it was originally run.
+pest_executable here can include the path to the executable if it's not already
+in the system's PATH.
+
+"--export-xls" is used to export contents of goPESTslaves.json into spreadsheet
+file goPESTslaves.xls.  This command can be used alone as it will load existing
+goPESTslaves.json.
+
+This command runs locally, and does not utilise any slurm/srun/queue facilities.
+
+This message can be printed with the "--help" argument.  It is possible to
+include more than one option at a time.
+'''
+
+def check_slaves_cli(argv=[]):
+    if '--help' in argv or len(argv) <= 1:
+        print(hlp)
+        exit(0)
+    else:
+        # overwrite slave_dirs
+        if '--dir' in argv:
+            iarg = argv.index('--dir') + 1
+            try:
+                spath = argv[iarg]
+            except IndexError:
+                raise Exception('--dir argument needs to be followed by path_to_slaves')
+        else:
+            spath = config['pest']['slave_dirs']
+        # overwrite PEST exe
+        if "--pest-exe" in argv:
+            iarg = argv.index('--pest-exe') + 1
+            try:
+                pexe = argv[iarg]
+            except IndexError:
+                raise Exception('--pest-exe argument needs to be followed by pest_executable')
+            # modify the shared module variable
+            config['pest']['dir'] = ''
+            config['pest']['executable'] = pexe
+        else:
+            # do nothing, keeps original setting
+            pass
+        # tasks
+        nopts, tasks = 0, []
+        if "--status" in argv:
+            nopts += 1
+            tasks.append('status')
+        if "--end-time" in argv:
+            nopts += 1
+            tasks.append('end-time')
+        if "--obj-fn" in argv:
+            nopts += 1
+            tasks.append('obj-fn')
+        xls = False
+        if "--export-xls" in argv:
+            nopts += 1
+            xls = True
+        if nopts == 0:
+            print(hlp)
+            print('Please specify at least one task to perform.')
+            exit(0)
+
+    start_time = time.time()
+    if not all([os.path.exists(spath), os.path.isdir(spath)]):
+        raise Exception('Specified path_to_slaves needs to be a valid directory: %s' % spath)
+
+    task_fn = {
+        'end-time': check_sim_ends,
+        'status': check_run_status,
+        'obj-fn': get_obj_fn,
+    }
+
+    slave_paths = sorted([p for p in glob.glob(os.path.join(spath, '*')) if os.path.isdir(p)])
+    slave_names = [os.path.basename(sp) for sp in slave_paths]
+
+    ncpu = max(1, cpu_count() - 1)
+    print('Starting %i workers to process %i slaves' % (ncpu, len(slave_paths)))
+    pool = Pool(ncpu, initializer=init_slave, initargs=(config, runtime))
+
+    fout = 'goPESTslaves.json'
+    if os.path.exists(fout):
+        with open(fout, 'r') as f:
+            data = json.load(f)
+    else:
+        data = {}
+
+    for task in tasks:
+        print('Checking %s...' % task)
+        results = pool.map(task_fn[task], slave_paths)
+        data = nested_dict_update(data, dict(zip(slave_names, results)))
+
+    with open(fout, 'w') as f:
+        json.dump(data, f, indent=4, sort_keys=True)
+
+    if xls:
+        export_xls(data)
+
+    print('Finished after %f seconds' % (time.time() - start_time))
+
+if __name__ == '__main__':
+    check_slaves_cli(sys.argv)

gopest/commands.py

@@ -0,0 +1,80 @@
+import sys
+
+from gopest import __version__
+
+title = """
+goPEST - Interfacing PEST with Waiwera and (AU)TOUGH2 simulators
+"""
+
+version = """Version: (%s)
+""" % __version__
+
+hlp = """
+Usage: gopest COMMAND [ARGUMENTS]
+
+Supported COMMANDs:
+    help
+    init [--no-copy][--no-par][--no-obs]    (make_case_pst)
+    submit                                  (submit_beopest)
+    run                                     (run_beopest)
+    par                                     (goPESTpar)
+    obs                                     (goPESTobs)
+    run-pest-model                          (pest_model)
+    run-forward                             (run_ns_pr)
+    save-iter-files                         (rename_latest_files)
+    check-slaves                            (check_slaves)
+
+Important files for goPEST to work:
+    goPESTconfig.toml
+        This is the main configuration file.
+    goPESTpar.list
+        This controls model parameters included for PEST.
+    goPESTobs.list
+        This controls observations for PEST.
+
+University of Auckland, 2012, 2022
+"""
+
+def gopest_cli():
+    print(title)
+    argc = len(sys.argv)
+    if argc < 2:
+        print(version + hlp)
+    else:
+        if sys.argv[1] == 'help':
+            print(version + hlp)
+        else:
+            # NOTE loading gopest.common checks goPESTconfig.toml
+            import gopest.common
+            import gopest.par
+            import gopest.obs
+            import gopest.pest_model
+            import gopest.run_ns_pr
+            import gopest.submit_beopest
+            import gopest.run_beopest
+            import gopest.make_case_pst
+            import gopest.rename_latest_files
+            import gopest.check_slaves
+            cmds = {
+                'par': gopest.par.goPESTpar,
+                'obs': gopest.obs.goPESTobs,
+                'run-pest-model': gopest.pest_model.main_cli,
+                'run-forward': gopest.run_ns_pr.main_cli,
+                'submit': gopest.submit_beopest.submit_cli,
+                'run': gopest.run_beopest.run_cli,
+                'init': gopest.make_case_pst.make_case_cli,
+                'save-iter-files': gopest.rename_latest_files.rename_latest_files,
+                'check-slaves': gopest.check_slaves.check_slaves_cli,
+            }
+            if sys.argv[1] not in cmds:
+                print(version + hlp)
+                print('Error! COMMAND not recognised.')
+                exit(1)
+            cmds[sys.argv[1]](sys.argv[1:])
+
+"""
+- NOTE good reference on designing CLI command names:
+  https://smallstep.com/blog/the-poetics-of-cli-command-names/
+
+- TODO it's probably not necessary to have so many commands
+"""