rda-python-dsquasar 2.0.11__tar.gz → 3.0.1__tar.gz

rda_python_dsquasar-3.0.1/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.4
+Name: rda_python_dsquasar
+Version: 3.0.1
+Summary: RDA Python package to backup and recover RDA data archives to and from GLOBUS Quasar backup server
+Author-email: Zaihua Ji <zji@ucar.edu>
+Project-URL: Homepage, https://github.com/NCAR/rda-python-dsquasar
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 5 - Production/Stable
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rda_python_common
+Requires-Dist: rda_python_dsarch
+Dynamic: license-file
+
+# rda_python_dsquasar
+
+RDA Python package to back up and recover RDA data archives to and from the
+GLOBUS Quasar backup server.
+
+## Programs
+
+The package installs the following command-line utilities, all running as the
+current user:
+
+- `dsquasar` — back up and restore RDA dataset archives via Quasar
+- `tacctar` — create tar bundles for Quasar archival
+- `taccrec` — recover content from Quasar tar bundles
+
+Run any program with `--help` (or `-h`) for full usage details.
+
+## Environment setup
+
+Create a Python environment first; package installs in the next section run
+inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
+
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-dsquasar
+
+Pick whichever install mode fits your workflow.  All variants pull in the
+transitive dependencies (`rda_python_common`, `rda_python_dsarch`)
+automatically.
+
+For local development, clone this repo alongside your project and install it
+in editable mode so that changes are picked up without re-installing:
+
+```bash
+git clone https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+For a regular (non-editable) install from a checkout:
+
+```bash
+pip install /path/to/rda-python-dsquasar
+```
+
+For a production install on a system that uses the published distribution:
+
+```bash
+pip install rda_python_dsquasar
+```

rda_python_dsquasar-3.0.1/README.md

@@ -0,0 +1,70 @@
+# rda_python_dsquasar
+
+RDA Python package to back up and recover RDA data archives to and from the
+GLOBUS Quasar backup server.
+
+## Programs
+
+The package installs the following command-line utilities, all running as the
+current user:
+
+- `dsquasar` — back up and restore RDA dataset archives via Quasar
+- `tacctar` — create tar bundles for Quasar archival
+- `taccrec` — recover content from Quasar tar bundles
+
+Run any program with `--help` (or `-h`) for full usage details.
+
+## Environment setup
+
+Create a Python environment first; package installs in the next section run
+inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
+
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-dsquasar
+
+Pick whichever install mode fits your workflow.  All variants pull in the
+transitive dependencies (`rda_python_common`, `rda_python_dsarch`)
+automatically.
+
+For local development, clone this repo alongside your project and install it
+in editable mode so that changes are picked up without re-installing:
+
+```bash
+git clone https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+For a regular (non-editable) install from a checkout:
+
+```bash
+pip install /path/to/rda-python-dsquasar
+```
+
+For a production install on a system that uses the published distribution:
+
+```bash
+pip install rda_python_dsquasar
+```

{rda_python_dsquasar-2.0.11 → rda_python_dsquasar-3.0.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rda_python_dsquasar"
-version = "2.0.11"
+version = "3.0.1"
 authors = [
   { name="Zaihua Ji", email="zji@ucar.edu" },
 ]

{rda_python_dsquasar-2.0.11 → rda_python_dsquasar-3.0.1}/src/rda_python_dsquasar/ds_quasar.py

@@ -91,7 +91,7 @@ TFCOUNT = 100          # if file count is greater, use MINSIZE for tar file
 SUBLMTS = 2000        # file count limit for a sub-group
 
 PGBACK = {
-   'workdir' : "{}/{}/quasar_backup".format(PgLOG.PGLOG['GDEXWORK'], PgLOG.PGLOG['GDEXUSER']),
+   'workdir' : "{}/{}/quasar_backup".format(PgLOG.PGLOG['GDEXWORK'], PgLOG.PGLOG['COMMONUSER']),
    'mproc' : 1,
    'action' : CTACTS,
    'chgdays' : 0,
@@ -1092,7 +1092,7 @@ def evaluate_file_stat(dsid, cate, pgrec):
    chome = PgLOG.PGLOG['DECSHOME'] if cate == 's' else PgLOG.PGLOG['DSDHOME']
 
    if pgrec['locflag'] == 'O':
-      bucket = 'gdex-decsdata' if cate == 's' else PgLOG.PGLOC['OBJCTBKT']
+      bucket = 'gdex-decsdata' if cate == 's' else PgLOG.PGLOG['OBJCTBKT']
       ofile = PgLOG.join_paths(dsid, fname)
       info = PgFile.check_object_file(ofile, bucket, 1, ERRACT)
       if not info: return False

{rda_python_dsquasar-2.0.11 → rda_python_dsquasar-3.0.1}/src/rda_python_dsquasar/dsquasar.py

@@ -85,7 +85,7 @@ class DsQuasar(PgCMD, PgSplit):
       self.TFCOUNT = 100          # if file count is greater, use self.MINSIZE for tar file
       self.SUBLMTS = 2000         # file count limit for a sub-group
       self.PGBACK = {
-         'workdir' : "{}/{}/quasar_backup".format(self.PGLOG['GDEXWORK'], self.PGLOG['GDEXUSER']),
+         'workdir' : "{}/{}/quasar_backup".format(self.PGLOG['GDEXWORK'], self.PGLOG['COMMONUSER']),
          'mproc' : 1,
          'action' : self.CTACTS,
          'chgdays' : 0,
@@ -920,7 +920,7 @@ class DsQuasar(PgCMD, PgSplit):
       fname = pgrec[tname]
       chome = self.PGLOG['DECSHOME'] if cate == 's' else self.PGLOG['DSDHOME']
       if pgrec['locflag'] == 'O':
-         bucket = 'gdex-decsdata' if cate == 's' else self.PGLOC['OBJCTBKT']
+         bucket = 'gdex-decsdata' if cate == 's' else self.PGLOG['OBJCTBKT']
          ofile = self.join_paths(dsid, fname)
          info = self.check_object_file(ofile, bucket, 1, self.ERRACT)
          if not info: return False

{rda_python_dsquasar-2.0.11 → rda_python_dsquasar-3.0.1}/src/rda_python_dsquasar/dstacc.py

@@ -81,7 +81,7 @@ class DsTACC(PgCMD, PgSplit):
       self.ONESIZE = 20*self.PGLOG['ONEGBS']  # 20GB, minimal file size to tar a single file 
       self.TFCOUNT = 1000000      # if file count is greater, use self.MINSIZE for tar file
       self.PGBACK = {
-         'workdir' : "{}/{}/tacc_backup".format(self.PGLOG['GDEXWORK'], self.PGLOG['GDEXUSER']),
+         'workdir' : "{}/{}/tacc_backup".format(self.PGLOG['GDEXWORK'], self.PGLOG['COMMONUSER']),
          'mproc' : 1,
          'action' : self.CTACTS,
          'chgdays' : 0,
@@ -857,7 +857,7 @@ class DsTACC(PgCMD, PgSplit):
       fname = pgrec[tname]
       chome = self.PGLOG['DECSHOME'] if cate == 's' else self.PGLOG['DSDHOME']
       if pgrec['locflag'] == 'O':
-         bucket = 'gdex-decsdata' if cate == 's' else self.PGLOC['OBJCTBKT']
+         bucket = 'gdex-decsdata' if cate == 's' else self.PGLOG['OBJCTBKT']
          ofile = self.join_paths(dsid, fname)
          info = self.check_object_file(ofile, bucket, 1, self.ERRACT)
          if not info: return False

{rda_python_dsquasar-2.0.11 → rda_python_dsquasar-3.0.1}/src/rda_python_dsquasar/taccrec.py

@@ -5,6 +5,8 @@ import argparse
 import psycopg2
 import hashlib
 from datetime import datetime
+import time
+from psycopg2 import OperationalError
 
 def get_tar_summary_and_details(member_list_path):
     details = []
@@ -17,7 +19,7 @@ def get_tar_summary_and_details(member_list_path):
         with open(member_list_path, 'r') as f:
             for idx, line in enumerate(f):
                 line = line.strip()
-                if not line or line.startswith('-rw') is False:
+                if not line:
                     continue
                 parts = line.split()
                 if len(parts) < 6:
@@ -68,60 +70,82 @@ def get_tar_summary_and_details(member_list_path):
         'dsid': dsid
     }, member_details
 
+def connect_with_retry(db_params, max_retries=1, delay=2):
+    attempt = 0
+    while attempt <= max_retries:
+        try:
+            return psycopg2.connect(**db_params)
+        except OperationalError as e:
+            if 'server closed the connection unexpectedly' in str(e) and attempt < max_retries:
+                time.sleep(delay)
+                attempt += 1
+                continue
+            raise
+
 def insert_tfile_row(summary, db_params, extra, update_on_conflict=False, member_details=None):
     table_name = 'dssdb.tfile'
-    conn = psycopg2.connect(**db_params)
-    cur = conn.cursor()
-    columns = [
-        'tfile', 'data_size', 'wcount', 'date_created', 'time_created',
-        'date_modified', 'time_modified', 'file_format', 'status',
-        'uid', 'dsid', 'data_format', 'disp_order', 'dsids', 'note'
-    ]
-    values = [
-        summary['tfile'], summary['data_size'], summary['wcount'],
-        summary['date_created'], summary['time_created'],
-        summary['date_modified'], summary['time_modified'],
-        'tar', 'T',
-        extra.get('uid'), extra.get('dsid'), extra.get('data_format'),
-        extra.get('disp_order'), extra.get('dsids'), extra.get('note')
-    ]
-    placeholders = ','.join(['%s'] * len(columns))
-    sql = f"INSERT INTO {table_name} ({', '.join(columns)}) VALUES ({placeholders})"
-    if update_on_conflict:
-        update_cols = [col for col in columns if col != 'tfile']
-        set_clause = ', '.join([f"{col}=EXCLUDED.{col}" for col in update_cols])
-        sql += f" ON CONFLICT (tfile) DO UPDATE SET {set_clause}"
-    try:
-        cur.execute(sql, values)
-        conn.commit()
-        # Retrieve tid for the just-inserted tfile row
-        cur.execute(f"SELECT tid FROM {table_name} WHERE tfile=%s", (summary['tfile'],))
-        row = cur.fetchone()
-        tid = row[0] if row else None
-        # Update wfile tables if member_details provided
-        if member_details and tid is not None:
-            for m in member_details:
-                name = m['name']
-                if '/' in name:
-                    cdsid, wfile = name.split('/', 1)
-                else:
-                    cdsid, wfile = name, ''
-                wfile_table = f"dssdb.wfile_{cdsid}"
-                # Check if table exists
-                cur.execute("SELECT to_regclass(%s)", (wfile_table,))
-                exists = cur.fetchone()[0]
-                if not exists:
-                    continue
-                # Update tid if record exists
-                cur.execute(f"UPDATE {wfile_table} SET tid=%s WHERE wfile=%s", (tid, wfile))
-        conn.commit()
-    except Exception as e:
-        print(f"Database error: {e}")
-        conn.rollback()
-        raise
-    finally:
-        cur.close()
-        conn.close()
+    attempt = 0
+    max_retries = 1
+    while attempt <= max_retries:
+        try:
+            conn = connect_with_retry(db_params)
+            cur = conn.cursor()
+            columns = [
+                'tfile', 'data_size', 'wcount', 'date_created', 'time_created',
+                'date_modified', 'time_modified', 'file_format', 'status',
+                'uid', 'dsid', 'data_format', 'disp_order', 'dsids', 'note'
+            ]
+            values = [
+                summary['tfile'], summary['data_size'], summary['wcount'],
+                summary['date_created'], summary['time_created'],
+                summary['date_modified'], summary['time_modified'],
+                'tar', 'T',
+                extra.get('uid'), extra.get('dsid'), extra.get('data_format'),
+                extra.get('disp_order'), extra.get('dsids'), extra.get('note')
+            ]
+            placeholders = ','.join(['%s'] * len(columns))
+            sql = f"INSERT INTO {table_name} ({', '.join(columns)}) VALUES ({placeholders})"
+            if update_on_conflict:
+                update_cols = [col for col in columns if col != 'tfile']
+                set_clause = ', '.join([f"{col}=EXCLUDED.{col}" for col in update_cols])
+                sql += f" ON CONFLICT (tfile) DO UPDATE SET {set_clause}"
+            cur.execute(sql, values)
+            conn.commit()
+            cur.execute(f"SELECT tid FROM {table_name} WHERE tfile=%s", (summary['tfile'],))
+            row = cur.fetchone()
+            tid = row[0] if row else None
+            if member_details and tid is not None:
+                for m in member_details:
+                    name = m['name']
+                    if '/' in name:
+                        cdsid, wfile = name.split('/', 1)
+                    else:
+                        cdsid, wfile = name, ''
+                    wfile_table = f"dssdb.wfile_{cdsid}"
+                    cur.execute("SELECT to_regclass(%s)", (wfile_table,))
+                    exists = cur.fetchone()[0]
+                    if not exists:
+                        continue
+                    cur.execute(f"UPDATE {wfile_table} SET tid=%s WHERE wfile=%s", (tid, wfile))
+            conn.commit()
+            cur.close()
+            conn.close()
+            break
+        except Exception as e:
+            if 'server closed the connection unexpectedly' in str(e) and attempt < max_retries:
+                attempt += 1
+                time.sleep(2)
+                continue
+            print(f"Database error: {e}")
+            try:
+                cur.close()
+            except:
+                pass
+            try:
+                conn.close()
+            except:
+                pass
+            raise
 
 def get_uid_from_logname(db_params):
     import getpass
@@ -130,17 +154,26 @@ def get_uid_from_logname(db_params):
         logname = os.getlogin()
     except Exception:
         logname = os.environ.get('USER') or getpass.getuser()
-    conn = psycopg2.connect(**db_params)
-    cur = conn.cursor()
-    # Ensure logname is quoted as a string in the query
-    cur.execute("SELECT userno FROM dssdb.dssgrp WHERE logname=%s LIMIT 1", (str(logname),))
-    row = cur.fetchone()
-    cur.close()
-    conn.close()
-    if row:
-        return row[0]
-    else:
-        raise ValueError(f"User logname '{logname}' not found in dssdb.dssgrp table.")
+    attempt = 0
+    max_retries = 1
+    while attempt <= max_retries:
+        try:
+            conn = connect_with_retry(db_params)
+            cur = conn.cursor()
+            cur.execute("SELECT userno FROM dssdb.dssgrp WHERE logname=%s LIMIT 1", (str(logname),))
+            row = cur.fetchone()
+            cur.close()
+            conn.close()
+            if row:
+                return row[0]
+            else:
+                raise ValueError(f"User logname '{logname}' not found in dssdb.dssgrp table.")
+        except Exception as e:
+            if 'server closed the connection unexpectedly' in str(e) and attempt < max_retries:
+                attempt += 1
+                time.sleep(2)
+                continue
+            raise
 
 def main():
     parser = argparse.ArgumentParser(description='Insert tar file summary into tfile table.')

rda_python_dsquasar-3.0.1/src/rda_python_dsquasar.egg-info/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.4
+Name: rda_python_dsquasar
+Version: 3.0.1
+Summary: RDA Python package to backup and recover RDA data archives to and from GLOBUS Quasar backup server
+Author-email: Zaihua Ji <zji@ucar.edu>
+Project-URL: Homepage, https://github.com/NCAR/rda-python-dsquasar
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 5 - Production/Stable
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rda_python_common
+Requires-Dist: rda_python_dsarch
+Dynamic: license-file
+
+# rda_python_dsquasar
+
+RDA Python package to back up and recover RDA data archives to and from the
+GLOBUS Quasar backup server.
+
+## Programs
+
+The package installs the following command-line utilities, all running as the
+current user:
+
+- `dsquasar` — back up and restore RDA dataset archives via Quasar
+- `tacctar` — create tar bundles for Quasar archival
+- `taccrec` — recover content from Quasar tar bundles
+
+Run any program with `--help` (or `-h`) for full usage details.
+
+## Environment setup
+
+Create a Python environment first; package installs in the next section run
+inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
+
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-dsquasar
+
+Pick whichever install mode fits your workflow.  All variants pull in the
+transitive dependencies (`rda_python_common`, `rda_python_dsarch`)
+automatically.
+
+For local development, clone this repo alongside your project and install it
+in editable mode so that changes are picked up without re-installing:
+
+```bash
+git clone https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-dsquasar.git
+cd rda-python-dsquasar
+pip install -e .
+```
+
+For a regular (non-editable) install from a checkout:
+
+```bash
+pip install /path/to/rda-python-dsquasar
+```
+
+For a production install on a system that uses the published distribution:
+
+```bash
+pip install rda_python_dsquasar
+```

rda_python_dsquasar-2.0.11/PKG-INFO

@@ -1,18 +0,0 @@
-Metadata-Version: 2.4
-Name: rda_python_dsquasar
-Version: 2.0.11
-Summary: RDA Python package to backup and recover RDA data archives to and from GLOBUS Quasar backup server
-Author-email: Zaihua Ji <zji@ucar.edu>
-Project-URL: Homepage, https://github.com/NCAR/rda-python-dsquasar
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Development Status :: 5 - Production/Stable
-Requires-Python: >=3.7
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: rda_python_common
-Requires-Dist: rda_python_dsarch
-Dynamic: license-file
-
-RDA python package to backup RDA dataset data onto Globus Quasar Server.

rda_python_dsquasar-2.0.11/README.md

@@ -1 +0,0 @@
-RDA python package to backup RDA dataset data onto Globus Quasar Server.

rda_python_dsquasar-2.0.11/src/rda_python_dsquasar.egg-info/PKG-INFO

@@ -1,18 +0,0 @@
-Metadata-Version: 2.4
-Name: rda_python_dsquasar
-Version: 2.0.11
-Summary: RDA Python package to backup and recover RDA data archives to and from GLOBUS Quasar backup server
-Author-email: Zaihua Ji <zji@ucar.edu>
-Project-URL: Homepage, https://github.com/NCAR/rda-python-dsquasar
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Development Status :: 5 - Production/Stable
-Requires-Python: >=3.7
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: rda_python_common
-Requires-Dist: rda_python_dsarch
-Dynamic: license-file
-
-RDA python package to backup RDA dataset data onto Globus Quasar Server.