rda-python-miscs 2.0.15__tar.gz → 3.0.0__tar.gz

rda_python_miscs-3.0.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.4
+Name: rda_python_miscs
+Version: 3.0.0
+Summary: RDA Python package to hold RDA miscellaneous utility programs
+Author-email: Zaihua Ji <zji@ucar.edu>
+Project-URL: Homepage, https://github.com/NCAR/rda-python-miscs
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 5 - Production/Stable
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rda_python_common
+Requires-Dist: rda_python_setuid
+Dynamic: license-file
+
+# rda_python_miscs
+
+RDA Python package to hold miscellaneous utility programs for the
+[NSF NCAR Geoscience Data Exchange (GDEX)](https://gdex.ucar.edu).
+
+## Programs
+
+The package provides two categories of programs:
+
+**Run as current user (no setuid required):**
+
+| Command | Alias | Description |
+|---------|-------|-------------|
+| `bashqsub` | | Submit a job as a bash batch job on a PBS node via qsub |
+| `tcshqsub` | | Submit a job as a tcsh batch job on a PBS node via qsub |
+| `rdasub` | `gdexsub` | Submit a command as a nohup background process on the local machine |
+| `pgwget` | | Download remote files by root name pattern, combining downloaded parts into a single file |
+| `gdexls` | | List local files and directories with matching metadata from the GDEX database |
+| `rdaps` | `gdexps` | Show process status for local or PBS batch processes, with filtering by PID, owner, or name |
+| `rdazip` | `gdexzip` | Compress or uncompress files using a supported format |
+| `rdaown` | `gdexown` | Change ownership of files and directories to rdadata (must be run as root) |
+| `pgrst` | | Convert .usg files to RST and push to gdex-docs-* repos on GitHub for readthedocs.io |
+
+**Run as gdexdata via setuid (requires setup below):**
+
+| Command | Alias | Connector script | Description |
+|---------|-------|-----------------|-------------|
+| `rdacp` | `gdexcp` | `setuid_rdacp` / `setuid_gdexcp` | Copy files and directories across local, remote, Object Store, or Globus endpoints |
+| `rdakill` | `gdexkill` | `setuid_rdakill` / `setuid_gdexkill` | Kill local processes and their children, or cancel PBS batch jobs |
+| `rdamod` | `gdexmod` | `setuid_rdamod` / `setuid_gdexmod` | Change permission modes for files and directories owned by rdadata |
+
+## Environment setup
+
+Create a Python environment first; package installs in the next section run
+inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
+
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-miscs
+
+Pick whichever install mode fits your workflow.  All four pull in the
+transitive dependencies (`rda_python_common`, `rda_python_setuid`)
+automatically.
+
+For local development, clone this repo alongside your project and install it
+in editable mode so that changes are picked up without re-installing:
+
+```bash
+git clone https://github.com/NCAR/rda-python-miscs.git
+cd rda-python-miscs
+pip install -e .
+```
+
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-miscs.git
+cd rda-python-miscs
+pip install -e .
+```
+
+For a regular (non-editable) install from a checkout:
+
+```bash
+pip install /path/to/rda-python-miscs
+```
+
+For a production install on a system that uses the published distribution:
+
+```bash
+pip install rda_python_miscs
+```
+
+## Setuid Setup
+
+The setuid programs (`rdacp`, `rdakill`, `rdamod` and their `gdex*` aliases)
+execute as the common user `PGLOG['COMMONUSER']` (default `gdexdata`) via
+the `rda_python_setuid` mechanism, which is pulled in automatically as a
+dependency.  After `pip install` above, choose one of the wiring options
+below.
+
+### Full setuid install (requires sudo access to COMMONUSER)
+
+Run these steps once per environment:
+
+```bash
+# 1. Compile the pywrapper C binary (once per environment):
+pywrapper-install -c|--compile -n|--username gdexdata
+
+# 2. Wire up all setuid programs in one pass:
+pywrapper-install -l|--link all
+
+# 3. Optionally, install a pgstart_<loginname> binary so <loginname> (any
+#    user in the same group as PGLOG['COMMONUSER']) can run commands as
+#    themselves.  Run either by PGLOG['ADMINUSER'] (default zji, if it has
+#    'sudo -u <loginname>'), or by <loginname> directly:
+pywrapper-install -p|--pgstart -n|--username <loginname>
+```
+
+`pywrapper-install` with no arguments displays the full user guide.
+
+### Simple install (no sudo required, runs as current user)
+
+Users who do not need the setuid mechanism can create direct symlinks instead:
+
+```bash
+pywrapper-install -l|--link all -s|--simple
+```
+
+This creates `bin/<name> -> bin/setuid_<name>` for every setuid program and
+they run as the current user with no privilege change.
+
+### Update an existing installation (no sudo required)
+
+When the package is upgraded and a new `pywrapper.c` is bundled, recompile and
+reinstall all setuid binaries using the existing `pgstart_*` binaries:
+
+```bash
+pywrapper-install -u|--update
+```
+
+### Setup guide
+
+The shared setuid setup guide is shown automatically if any `setuid_*`
+connector script is invoked directly before the setuid wrapper has been
+configured.

rda_python_miscs-3.0.0/README.md

@@ -0,0 +1,139 @@
+# rda_python_miscs
+
+RDA Python package to hold miscellaneous utility programs for the
+[NSF NCAR Geoscience Data Exchange (GDEX)](https://gdex.ucar.edu).
+
+## Programs
+
+The package provides two categories of programs:
+
+**Run as current user (no setuid required):**
+
+| Command | Alias | Description |
+|---------|-------|-------------|
+| `bashqsub` | | Submit a job as a bash batch job on a PBS node via qsub |
+| `tcshqsub` | | Submit a job as a tcsh batch job on a PBS node via qsub |
+| `rdasub` | `gdexsub` | Submit a command as a nohup background process on the local machine |
+| `pgwget` | | Download remote files by root name pattern, combining downloaded parts into a single file |
+| `gdexls` | | List local files and directories with matching metadata from the GDEX database |
+| `rdaps` | `gdexps` | Show process status for local or PBS batch processes, with filtering by PID, owner, or name |
+| `rdazip` | `gdexzip` | Compress or uncompress files using a supported format |
+| `rdaown` | `gdexown` | Change ownership of files and directories to rdadata (must be run as root) |
+| `pgrst` | | Convert .usg files to RST and push to gdex-docs-* repos on GitHub for readthedocs.io |
+
+**Run as gdexdata via setuid (requires setup below):**
+
+| Command | Alias | Connector script | Description |
+|---------|-------|-----------------|-------------|
+| `rdacp` | `gdexcp` | `setuid_rdacp` / `setuid_gdexcp` | Copy files and directories across local, remote, Object Store, or Globus endpoints |
+| `rdakill` | `gdexkill` | `setuid_rdakill` / `setuid_gdexkill` | Kill local processes and their children, or cancel PBS batch jobs |
+| `rdamod` | `gdexmod` | `setuid_rdamod` / `setuid_gdexmod` | Change permission modes for files and directories owned by rdadata |
+
+## Environment setup
+
+Create a Python environment first; package installs in the next section run
+inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
+
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-miscs
+
+Pick whichever install mode fits your workflow.  All four pull in the
+transitive dependencies (`rda_python_common`, `rda_python_setuid`)
+automatically.
+
+For local development, clone this repo alongside your project and install it
+in editable mode so that changes are picked up without re-installing:
+
+```bash
+git clone https://github.com/NCAR/rda-python-miscs.git
+cd rda-python-miscs
+pip install -e .
+```
+
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-miscs.git
+cd rda-python-miscs
+pip install -e .
+```
+
+For a regular (non-editable) install from a checkout:
+
+```bash
+pip install /path/to/rda-python-miscs
+```
+
+For a production install on a system that uses the published distribution:
+
+```bash
+pip install rda_python_miscs
+```
+
+## Setuid Setup
+
+The setuid programs (`rdacp`, `rdakill`, `rdamod` and their `gdex*` aliases)
+execute as the common user `PGLOG['COMMONUSER']` (default `gdexdata`) via
+the `rda_python_setuid` mechanism, which is pulled in automatically as a
+dependency.  After `pip install` above, choose one of the wiring options
+below.
+
+### Full setuid install (requires sudo access to COMMONUSER)
+
+Run these steps once per environment:
+
+```bash
+# 1. Compile the pywrapper C binary (once per environment):
+pywrapper-install -c|--compile -n|--username gdexdata
+
+# 2. Wire up all setuid programs in one pass:
+pywrapper-install -l|--link all
+
+# 3. Optionally, install a pgstart_<loginname> binary so <loginname> (any
+#    user in the same group as PGLOG['COMMONUSER']) can run commands as
+#    themselves.  Run either by PGLOG['ADMINUSER'] (default zji, if it has
+#    'sudo -u <loginname>'), or by <loginname> directly:
+pywrapper-install -p|--pgstart -n|--username <loginname>
+```
+
+`pywrapper-install` with no arguments displays the full user guide.
+
+### Simple install (no sudo required, runs as current user)
+
+Users who do not need the setuid mechanism can create direct symlinks instead:
+
+```bash
+pywrapper-install -l|--link all -s|--simple
+```
+
+This creates `bin/<name> -> bin/setuid_<name>` for every setuid program and
+they run as the current user with no privilege change.
+
+### Update an existing installation (no sudo required)
+
+When the package is upgraded and a new `pywrapper.c` is bundled, recompile and
+reinstall all setuid binaries using the existing `pgstart_*` binaries:
+
+```bash
+pywrapper-install -u|--update
+```
+
+### Setup guide
+
+The shared setuid setup guide is shown automatically if any `setuid_*`
+connector script is invoked directly before the setuid wrapper has been
+configured.

{rda_python_miscs-2.0.15 → rda_python_miscs-3.0.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rda_python_miscs"
-version = "2.0.15"
+version = "3.0.0"
 authors = [
   { name="Zaihua Ji", email="zji@ucar.edu" },
 ]
@@ -36,13 +36,19 @@ pythonpath = [
 "bashqsub" = "rda_python_miscs.bashqsub:main"
 "tcshqsub" = "rda_python_miscs.tcshqsub:main"
 "rdasub" = "rda_python_miscs.rdasub:main"
+"gdexsub" = "rda_python_miscs.rdasub:main"
 "pgwget" = "rda_python_miscs.pgwget:main"
-"rdals" = "rda_python_miscs.rdals:main"
 "gdexls" = "rda_python_miscs.gdexls:main"
 "rdaps" = "rda_python_miscs.rdaps:main"
+"gdexps" = "rda_python_miscs.rdaps:main"
 "rdazip" = "rda_python_miscs.rdazip:main"
+"gdexzip" = "rda_python_miscs.rdazip:main"
 "rdaown" = "rda_python_miscs.rdaown:main"
-"rdacp.py" = "rda_python_miscs.rdacp:main"
-"rdakill.py" = "rda_python_miscs.rdakill:main"
-"rdamod.py" = "rda_python_miscs.rdamod:main"
+"gdexown" = "rda_python_miscs.rdaown:main"
+"setuid_rdacp" = "rda_python_miscs.rdacp:main"
+"setuid_gdexcp" = "rda_python_miscs.rdacp:main"
+"setuid_rdakill" = "rda_python_miscs.rdakill:main"
+"setuid_gdexkill" = "rda_python_miscs.rdakill:main"
+"setuid_rdamod" = "rda_python_miscs.rdamod:main"
+"setuid_gdexmod" = "rda_python_miscs.rdamod:main"
 "pgrst" = "rda_python_miscs.pg_rst:main"

{rda_python_miscs-2.0.15 → rda_python_miscs-3.0.0}/src/rda_python_miscs/bashqsub.py

@@ -7,7 +7,7 @@
 #            https://github.com/NCAR/rda-utility-programs.git
 #            2025-12-29 convert to class BashQsub
 #   Purpose: python script to submit a batch job on PBS node via bash script
-#    Github: https://github.com/NCAR/rda-pythn-miscs.git
+#    Github: https://github.com/NCAR/rda-python-miscs.git
 ##################################################################################
 import os
 import sys
@@ -16,8 +16,14 @@ from os import path as op
 from rda_python_common.pg_log import PgLOG
 
 class BashQsub(PgLOG):
+   """Submit a PBS batch job via a dynamically generated bash script using qsub.
+
+   Builds a bash script with PBS directives, module loads, and conda environment
+   activation, then submits it through the PBS qsub command.
+   """
 
    def __init__(self):
+      """Initialize BashQsub with default PBS resource settings and options."""
       super().__init__()
       self.DEFMODS = {
          'default': "ncarenv,netcdf,ncl,nco,cdo,conda,grib-util,wgrib2"
@@ -42,8 +48,15 @@ class BashQsub(PgLOG):
       self.gdexsub = self.BCHCMDS['PBS']
       self.args = None
 
-   # function to readparameters
+   # function to read parameters
    def read_parameters(self):
+      """Parse command-line arguments and populate PBS options and customized options.
+
+      Handles single-dash qsub options (e.g. -q, -A, -l) and long custom options
+      (-cmd, -cwd, -env, -mod, -res). Validates that the qsub command is available
+      and that a -cmd value is provided. Sets default log paths and job name if not
+      specified, and changes the working directory if -cwd is given.
+      """
       aname = 'bashqsub'
       pname = 'gdexqsub'
       self.set_help_path(__file__)
@@ -88,11 +101,12 @@ class BashQsub(PgLOG):
       if not self.SOPTIONS['e']: self.SOPTIONS['e'] = "{}/{}/".format(self.PGLOG['LOGPATH'], pname)
       if 'N' not in self.SOPTIONS: self.SOPTIONS['N'] = op.basename(self.coptions['cmd'])
       if self.coptions['cwd']:
-         if 's' in self.coptions['cwd']: self.coptions['cwd'] = self.replace_environments(self.coptions['cwd'], '', self.LGWNEX)
+         if '$' in self.coptions['cwd']: self.coptions['cwd'] = self.replace_environments(self.coptions['cwd'], '', self.LGWNEX)
          os.chdir(self.coptions['cwd'])
 
    # function to start actions
    def start_actions(self):
+      """Resolve the command path, build the bash script, and submit it via qsub."""
       cmd = self.valid_command(self.coptions['cmd'])
       if not cmd and not re.match(r'^/', self.coptions['cmd']): cmd = self.valid_command('./' + self.coptions['cmd'])
       if not cmd: self.pglog(self.coptions['cmd'] + ": Cannot find given command to run", self.LGWNEX)
@@ -105,6 +119,17 @@ class BashQsub(PgLOG):
    
    # build bash script to submit a PBS batch job
    def build_bash_script(self, cmd):
+      """Build and return a bash script string with PBS directives for the given command.
+
+      Sets HOME, sources system and conda profile scripts and the user's .bashrc,
+      loads modules, activates the conda environment, then runs the command.
+
+      Args:
+         cmd (str): The fully-resolved command (with arguments) to execute in the job.
+
+      Returns:
+         str: The complete bash batch script content.
+      """
       buf = "#!/usr/bin/bash\n\n"   # qsub starting bash script
       if 'l' in self.SOPTIONS: self.add_resources()
       # add options to bash script for qsub
@@ -128,8 +153,13 @@ class BashQsub(PgLOG):
       buf += "\necho {}\n{}\n\ndate\n".format(cmd, cmd)
       return buf
 
-   # check and add resource options 
+   # check and add resource options
    def add_resources(self):
+      """Parse -l option value into the RESOURCES dict and remove the raw -l entry.
+
+      Expects comma-separated name=value pairs (e.g. 'walltime=2:00:00,select=1:ncpus=4').
+      Logs an error if a token does not contain '='.
+      """
       for res in re.split(',', self.SOPTIONS['l']):
          ms = re.match(r'^([^=]+)=(.+)$', res)
          if ms:
@@ -140,6 +170,20 @@ class BashQsub(PgLOG):
 
    # add module loads for modules provided
    def add_modules(self, res, mods):
+      """Build and return module load/unload commands for the bash script.
+
+      Loads the default module set for the given reservation (or the 'default' set).
+      Additional modules in ``mods`` are appended; path-style entries (starting with
+      '/') use 'module use' instead of 'module load'.  Modules already in the default
+      set are skipped.  SWAPMODS entries trigger an unload before the new load.
+
+      Args:
+         res (str): Reservation name used to look up DEFMODS; falls back to 'default'.
+         mods (str): Comma-separated list of extra modules (or None).
+
+      Returns:
+         str: Shell commands to load/unload modules.
+      """
       mbuf = "\n"
       defmods = self.DEFMODS[res] if res in self.DEFMODS else self.DEFMODS['default']
       dmods = re.split(',', defmods)
@@ -163,6 +207,16 @@ class BashQsub(PgLOG):
 
    # set virtual machine libraries
    def set_vm_libs(self, res):
+      """Build and return conda/VM library activation commands for the bash script.
+
+      Looks up DEFLIBS for the given reservation (falls back to 'default').
+
+      Args:
+         res (str): Reservation name used to look up DEFLIBS; falls back to 'default'.
+
+      Returns:
+         str: Shell commands to activate virtual environment libraries, or '' if none.
+      """
       deflibs = self.DEFLIBS[res] if res in self.DEFLIBS else self.DEFLIBS['default']
       if not deflibs: return ''
       dlibs = re.split(',', deflibs)
@@ -171,8 +225,9 @@ class BashQsub(PgLOG):
          libbuf += dlib + "\n"
       return libbuf
 
-# main function to excecute this script
+# main function to execute this script
 def main():
+   """Entry point: instantiate BashQsub, parse arguments, run, and exit."""
    object = BashQsub()
    object.read_parameters()
    object.start_actions()

{rda_python_miscs-2.0.15 → rda_python_miscs-3.0.0}/src/rda_python_miscs/bashqsub.usg

@@ -10,9 +10,9 @@
             -o LOGPATH/gdexqsub/
             -e LOGPATH/gdexqsub/
             -A P43713000
-            -m a
-            -q gdex
-            -l walltime=6:00:00,select=1:node=1:mem=1gb
+            -m n
+            -q gdex@casper-pbs
+            -l walltime=6:00:00,select=1:ncpus=1:mem=1gb
 
       - Option -cwd, set the working directory for the Command to be executed. If
           it is not specified, it defaults to the current directory where qsub
@@ -33,18 +33,18 @@
 A bash script example:
 #!/usr/bin/bash
 
-#PBS -o /gpfs/u/home/gdexdata/dssdb/log/gdexqsub/
-#PBS -e /gpfs/u/home/gdexdata/dssdb/log/gdexqsub/
+#PBS -o /glade/u/home/gdexdata/dssdb/log/gdexqsub/
+#PBS -e /glade/u/home/gdexdata/dssdb/log/gdexqsub/
 #PBS -A P43713000
 #PBS -q gdex@casper-pbs
 #PBS -m n
 #PBS -N dsrqst
 #PBS -l walltime=1:00:00
 #PBS -l select=1:ncpus=1:mem=1gb
-export HOME=/gpfs/u/home/zji
+export HOME=/glade/u/home/zji
 source /etc/profile.d/z00_modules.sh
 source /glade/u/apps/opt/conda/etc/profile.d/conda.sh
-source /gpfs/u/home/zji/.bashrc
+source /glade/u/home/zji/.bashrc
 pwd; hostname; date
 
 module load ncarenv