slurm2sql 0.9.2__tar.gz → 0.9.4__tar.gz

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/.github/workflows/pyrelease.yaml

@@ -31,7 +31,7 @@ on:
   # For tag-based (instead of Github-specific release-based):
   push:
     tags:
-      - '*'
+      - '*.*.*'
 
 jobs:
   build:

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/.github/workflows/pytest.yml

@@ -6,7 +6,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.7', '3.9', '3.x']
+        python-version: ['3.9', '3.11', '3.x']
     steps:
     - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }}

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: slurm2sql
-Version: 0.9.2
+Version: 0.9.4
 Summary: Import Slurm accounting database from sacct to sqlite3 database
 Keywords: slurm,sqlite3
 Author: Richard Darst
@@ -16,6 +16,7 @@ Classifier: Intended Audience :: System Administrators
 Classifier: Topic :: Database
 Classifier: Topic :: System :: Clustering
 Classifier: Topic :: System :: Distributed Computing
+License-File: LICENSE
 Requires-Dist: tabulate
 Requires-Dist: pytest ; extra == "test"
 Project-URL: Repository, https://github.com/NordicHPC/slurm2sql
@@ -275,44 +276,44 @@ them.  For other columns, check ``man sacct``.
     stripped out and give invalid data.  File an issue and this will
     be added.
 
-* ``ReqMem``: The raw slurm value in a format like "5Gn".  Instead of
-  parsing this, you probably want to use one of the other values below.
+* **Memory related**
 
-* ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,
-  either taken from ReqMem (if it matches) or computed (you might want
-  to check our logic if you rely on this).  In Slurm, you
-  can request memory either per-node or per-core, and this calculates
-  the other one for you.
+  * ``AllocMem``: The ``mem=`` value from ``AllocTRES`` field.  You
+    probably want to use this.
 
-* ``ReqMemType``: ``c`` if the user requested mem-per-core originally,
-  ``n`` if mem-per-node.  Extracted from ``ReqMem``.  Modern Slurm has
-  nothing here, and the column value is null.
+  * ``TotalMem``: The ``mem=`` value from ``TRESUsageInTot`` field.
+    You probably want to use this.
 
-* ``ReqMemRaw``: The numeric value of the ``ReqMem``, whether it is
-  ``c`` or ``n``.
+  * ``ReqMem``: The raw slurm value from the ReqMem column.
 
-* ``ReqGPU``: Number of GPUs requested.  Extracted from ``ReqTRES``.
+  * ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,
+    ``ReqMem`` / ``NNodes``.
 
-* GPU information.  These use values from the ``TRESUsageInAve``
+  * ``MemEff``: Computed ``TotalMem / AllocMem``.
+
+* **GPU information.**  These use values from the ``TRESUsageInAve``
   fields in modern Slurm
 
-  * ``GpuMem``: ``gres/gpumem``
+  * ``ReqGPU``: Number of GPUs requested.  Extracted from ``ReqTRES``.
+
+  * ``GpuMem``: ``gres/gpumem`` from ``TRESUsageInAve``
 
   * ``GpuUtil``: ``gres/gpuutil`` (fraction 0.0-1.0).
 
-  * ``NGpus``: Number of GPUs.  Should be the same as ``ReqGPU``, but
-    who knows.
+  * ``NGpus``: Number of GPUs from ``gres/gpu`` in ``AllocTRES``.
+    Should be the same as ``ReqGPU``, but who knows.
 
   * ``GpuUtilTot``, ``GpuMemTot``: like above but using the
     ``TRESUsageInTot`` sacct field.
 
-* ``MemEff``: This is null in the Slurm table now, since Slurm gives
-  ReqMem in allocations and memory used in steps.  The ``eff`` table
-  calculates this now.
+  * ``GpuEff``: ``gres/gpuutil`` (from ``TRESUsageInTot``) / (100 *
+    ``gres/gpu`` (from ``AllocTRES``).
 
 * ``CPUEff``: CPU efficiency (0.0-1.0).  All the same caveats as above
   apply: test before trusting.
 
+* And more, see the code for now.
+
 Quick reference of the other most important columns from the
 accounting database that are hardest to remember:
 
@@ -325,12 +326,11 @@ accounting database that are hardest to remember:
 
 The ``eff`` table adds the following:
 
-* ``CPUEff``: like CPUEff but for the whole job
+* ``CPUEff``: Highest CPUEff for any job step
 
-* ``MemEff``: Memory efficiency for the whole job (max(MaxRSS) /
-  ReqMem)
+* ``MemEff``: Highest MemEff for any job step
 
-* And more, see the code for now.
+* ``GpuEff``: Highest GpuEff for any job step
 
 
 

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/README.rst

@@ -252,44 +252,44 @@ them.  For other columns, check ``man sacct``.
     stripped out and give invalid data.  File an issue and this will
     be added.
 
-* ``ReqMem``: The raw slurm value in a format like "5Gn".  Instead of
-  parsing this, you probably want to use one of the other values below.
+* **Memory related**
 
-* ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,
-  either taken from ReqMem (if it matches) or computed (you might want
-  to check our logic if you rely on this).  In Slurm, you
-  can request memory either per-node or per-core, and this calculates
-  the other one for you.
+  * ``AllocMem``: The ``mem=`` value from ``AllocTRES`` field.  You
+    probably want to use this.
 
-* ``ReqMemType``: ``c`` if the user requested mem-per-core originally,
-  ``n`` if mem-per-node.  Extracted from ``ReqMem``.  Modern Slurm has
-  nothing here, and the column value is null.
+  * ``TotalMem``: The ``mem=`` value from ``TRESUsageInTot`` field.
+    You probably want to use this.
 
-* ``ReqMemRaw``: The numeric value of the ``ReqMem``, whether it is
-  ``c`` or ``n``.
+  * ``ReqMem``: The raw slurm value from the ReqMem column.
 
-* ``ReqGPU``: Number of GPUs requested.  Extracted from ``ReqTRES``.
+  * ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,
+    ``ReqMem`` / ``NNodes``.
 
-* GPU information.  These use values from the ``TRESUsageInAve``
+  * ``MemEff``: Computed ``TotalMem / AllocMem``.
+
+* **GPU information.**  These use values from the ``TRESUsageInAve``
   fields in modern Slurm
 
-  * ``GpuMem``: ``gres/gpumem``
+  * ``ReqGPU``: Number of GPUs requested.  Extracted from ``ReqTRES``.
+
+  * ``GpuMem``: ``gres/gpumem`` from ``TRESUsageInAve``
 
   * ``GpuUtil``: ``gres/gpuutil`` (fraction 0.0-1.0).
 
-  * ``NGpus``: Number of GPUs.  Should be the same as ``ReqGPU``, but
-    who knows.
+  * ``NGpus``: Number of GPUs from ``gres/gpu`` in ``AllocTRES``.
+    Should be the same as ``ReqGPU``, but who knows.
 
   * ``GpuUtilTot``, ``GpuMemTot``: like above but using the
     ``TRESUsageInTot`` sacct field.
 
-* ``MemEff``: This is null in the Slurm table now, since Slurm gives
-  ReqMem in allocations and memory used in steps.  The ``eff`` table
-  calculates this now.
+  * ``GpuEff``: ``gres/gpuutil`` (from ``TRESUsageInTot``) / (100 *
+    ``gres/gpu`` (from ``AllocTRES``).
 
 * ``CPUEff``: CPU efficiency (0.0-1.0).  All the same caveats as above
   apply: test before trusting.
 
+* And more, see the code for now.
+
 Quick reference of the other most important columns from the
 accounting database that are hardest to remember:
 
@@ -302,12 +302,11 @@ accounting database that are hardest to remember:
 
 The ``eff`` table adds the following:
 
-* ``CPUEff``: like CPUEff but for the whole job
+* ``CPUEff``: Highest CPUEff for any job step
 
-* ``MemEff``: Memory efficiency for the whole job (max(MaxRSS) /
-  ReqMem)
+* ``MemEff``: Highest MemEff for any job step
 
-* And more, see the code for now.
+* ``GpuEff``: Highest GpuEff for any job step
 
 
 

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/pyproject.toml

@@ -39,6 +39,8 @@ test = [
     slurm2sql = "slurm2sql:main"
     slurm2sql-sacct = "slurm2sql:sacct_cli"
     slurm2sql-seff = "slurm2sql:seff_cli"
+    sacct2 = "slurm2sql:sacct_cli"
+    seff2 = "slurm2sql:seff_cli"
 
 [project.urls]
 Repository = "https://github.com/NordicHPC/slurm2sql"

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/slurm2sql.py

@@ -18,7 +18,7 @@ import subprocess
 import sys
 import time
 
-__version__ = '0.9.2'
+__version__ = '0.9.4'
 
 LOG = logging.getLogger('slurm2sql')
 LOG.setLevel(logging.DEBUG)
@@ -383,6 +383,19 @@ class slurmGPUCount(linefunc):
         if m:
             return int(m.group(1))
 
+RE_TRES_GPU = re.compile(rf'\bgres/gpu=([^,]*)\b')
+RE_TRES_GPU_UTIL = re.compile(rf'\bgres/gpuutil=([^,]*)\b')
+class slurmGPUEff2(linefunc):
+    """Slurm GPU efficiency (using AllocTRES and TRESUsageInTot columns).
+    """
+    type = 'real'
+    @staticmethod
+    def calc(row):
+        m_used = RE_TRES_GPU_UTIL.search(row['TRESUsageInTot'])
+        m_alloc = RE_TRES_GPU.search(row['AllocTRES'])
+        if m_alloc and m_used:
+            return (float_metric(m_used.group(1)) / 100.) / float_metric(m_alloc.group(1))
+        return None
 
 # Job ID related stuff
 jobidonly_re = re.compile(r'[0-9]+')
@@ -467,6 +480,23 @@ class slurmMemEff(linefunc):
             raise ValueError('unknown memory type: %s'%reqmem_type)
         return mem_max / nodemem
 
+RE_TRES_MEM = re.compile(rf'\bmem=([^,]*)\b')
+class slurmMemEff2(linefunc):
+    """Slurm memory efficiency (using AllocTRES and TRESUsageInTot columns).
+
+    This *does* work in new enough Slurm.
+    """
+    # https://github.com/SchedMD/slurm/blob/master/contribs/seff/seff
+    type = 'real'
+    @staticmethod
+    def calc(row):
+        m_used = RE_TRES_MEM.search(row['TRESUsageInTot'])
+        m_alloc = RE_TRES_MEM.search(row['AllocTRES'])
+        if m_alloc and m_used:
+            return float_bytes(m_used.group(1)) / float_bytes(m_alloc.group(1))
+        return None
+
+
 class slurmCPUEff(linefunc):
     # This matches the seff tool currently:
     # https://github.com/SchedMD/slurm/blob/master/contribs/seff/seff
@@ -589,6 +619,9 @@ COLUMNS = {
     'MinCPUTask': nullstr,
 
     # Memory related
+    '_TotalMem': ExtractField('TotalMem', 'TRESUsageInTot', 'mem', float_bytes),
+    '_AllocMem': ExtractField('AllocMem', 'AllocTRES', 'mem', float_bytes),
+    '_MemEff': slurmMemEff2,            # Calculated from AllocTRES and TRESUsageInTot
     'ReqMem': float_bytes,              # Requested mem, value from slurm.  Sum across all nodes
     '_ReqMemNode': slurmMemNode,        # Mem per node, computed
     '_ReqMemCPU': slurmMemCPU,          # Mem per cpu, computed
@@ -598,7 +631,6 @@ COLUMNS = {
     'MaxRSSTask': nullstr,
     'MaxPages': int_metric,
     'MaxVMSize': slurmmem,
-    #'_MemEff': slurmMemEff,             # Slurm memory efficiency - see above for why this doesn't work
 
     # Disk related
     'AveDiskRead': int_bytes,
@@ -614,10 +646,10 @@ COLUMNS = {
     '_ReqGPUS': ExtractField('ReqGpus', 'ReqTRES', 'gres/gpu', float_metric),
     'Comment': nullstr_strip,           # Slurm Comment field (at Aalto used for GPU stats)
     #'_GPUMem': slurmGPUMem,             # GPU mem extracted from comment field
-    #'_GPUEff': slurmGPUEff,             # GPU utilization (0.0 to 1.0) extracted from comment field
+    '_GpuEff': slurmGPUEff2,             # GPU utilization (0.0 to 1.0) from AllocTRES()
     #'_NGPU': slurmGPUCount,             # Number of GPUs, extracted from comment field
     '_NGpus': ExtractField('NGpus', 'AllocTRES', 'gres/gpu', float_metric),
-    '_GpuUtil': ExtractField('GpuUtil', 'TRESUsageInAve', 'gres/gpuutil', float_metric, wrap=lambda x: x/100.),
+    '_GpuUtil': ExtractField('GpuUtil', 'TRESUsageInAve', 'gres/gpuutil', float_metric, wrap=lambda x: x/100.), # can be >100 for multi-GPU.
     '_GpuMem': ExtractField('GpuMem2', 'TRESUsageInAve', 'gres/gpumem', float_metric),
     '_GpuUtilTot': ExtractField('GpuUtilTot', 'TRESUsageInTot', 'gres/gpuutil', float_metric),
     '_GpuMemTot': ExtractField('GpuMemTot',   'TRESUsageInTot', 'gres/gpumem', float_metric),
@@ -671,7 +703,7 @@ def main(argv=sys.argv[1:], db=None, raw_sacct=None, csv_input=None):
         logging.lastResort.setLevel(logging.WARN)
     LOG.debug(args)
 
-    sacct_filter = process_sacct_filter(args, sacct_filter)
+    sacct_filter = args_to_sacct_filter(args, sacct_filter)
 
     # db is only given as an argument in tests (normally)
     if db is None:
@@ -864,10 +896,11 @@ def slurm2sql(db, sacct_filter=['-a'], update=False, jobs_only=False,
     db.execute('CREATE TABLE IF NOT EXISTS slurm (%s)'%create_columns)
     db.execute('CREATE TABLE IF NOT EXISTS meta_slurm_lastupdate (id INTEGER PRIMARY KEY, update_time REAL)')
     db.execute('CREATE VIEW IF NOT EXISTS allocations AS select * from slurm where JobStep is null;')
+    db.execute('CREATE VIEW IF NOT EXISTS steps AS select * from slurm where JobStep is not null;')
     db.execute('CREATE VIEW IF NOT EXISTS eff AS select '
                'JobIDnostep AS JobID, '
                'max(User) AS User, '
-               'max(Partition), '
+               'max(Partition) AS Partition, '
                'Account, '
                'State, '
                'Time, '
@@ -882,20 +915,23 @@ def slurm2sql(db, sacct_filter=['-a'], update=False, jobs_only=False,
                'max(cputime) AS cpu_s_reserved, '
                'max(totalcpu) AS cpu_s_used, '
                'max(ReqMemNode) AS MemReq, '
-               'max(ReqMemNode*Elapsed) AS mem_s_reserved, ' # highest of any job
+               'max(AllocMem) AS AllocMem, '
+               'max(TotalMem) AS TotalMem, '
                'max(MaxRSS) AS MaxRSS, '
-               'max(MaxRSS) / max(ReqMemNode) AS MemEff, '
+               'max(MemEff) AS MemEff, '
+               'max(AllocMem*Elapsed) AS mem_s_reserved, ' # highest of any job
                'max(NGpus) AS NGpus, '
                'max(NGpus)*max(Elapsed) AS gpu_s_reserved, '
                'max(NGpus)*max(Elapsed)*max(GPUutil) AS gpu_s_used, '
-               'max(GPUutil) AS GPUeff, '               # Individual job with highest use (check this)
+               #'max(GPUutil)/max(NGpus) AS GPUeff, '               # Individual job with highest use (check this)
+               'max(GPUEff) AS GPUeff, '               # Individual job with highest use (check this)
                'max(GPUMem) AS GPUMem, '
                'MaxDiskRead, '
                'MaxDiskWrite, '
                'sum(TotDiskRead) as TotDiskRead, '
                'sum(TotDiskWrite) as TotDiskWrite '
                'FROM slurm GROUP BY JobIDnostep')
-    db.execute('PRAGMA journal_mode = WAL;')
+    #db.execute('PRAGMA journal_mode = WAL;')
     db.commit()
     c = db.cursor()
 
@@ -946,7 +982,7 @@ def slurm2sql(db, sacct_filter=['-a'], update=False, jobs_only=False,
     return errors[0]
 
 
-def process_sacct_filter(args, sacct_filter):
+def args_to_sacct_filter(args, sacct_filter):
     """Generate sacct filter args in a standard way
 
     For example adding a --completed argument that translates into
@@ -958,8 +994,52 @@ def process_sacct_filter(args, sacct_filter):
     # Set for completed jobs.
     if getattr(args, 'completed', None):
         sacct_filter[:0] = ['--endtime=now', f'--state={COMPLETED_STATES}']
+    if getattr(args, 'user', None):
+        sacct_filter[:0] = [f'--user={args.user}']
+        # Set args.user to None.  We have already handled it here and
+        # it shouldn't be re-handled in the future SQL code (future
+        # SQL woludn't handle multiple users, for example).
+        args.user = None
+    if getattr(args, 'partition', None):
+        sacct_filter[:0] = [f'--partition={args.partition}']
+        args.partition = None
+    if getattr(args, 'running_at_time', None):
+        sacct_filter[:0] = [f'--start={args.running_at_time}', f'--end={args.running_at_time}', '--state=RUNNING' ]
+        args.running_at_time = None
     return sacct_filter
 
+def args_to_sql_where(args):
+    where = [ ]
+    if getattr(args, 'user', None):
+        where.append('and user=:user')
+    if getattr(args, 'partition', None):
+        where.append("and Partition like '%'||:partition||'%'")
+    return ' '.join(where)
+
+
+def import_or_open_db(args, sacct_filter, csv_input=None):
+    """Helper function to either open a DB or generate a new in-mem one from sacct
+
+    The `args` sholud be an argparse argument option.  This function
+    will look at its arguments and do what it says.  So, if you want
+    various features, you need to define these arguments in argparse:
+
+    db: filename of a database to open
+
+    """
+    if args.db:
+        db = sqlite3.connect(args.db)
+        if sacct_filter:
+            LOG.warn("Warning: reading from database.  Any sacct filters are ignored.")
+    else:
+        # Import fresh
+        sacct_filter = args_to_sacct_filter(args, sacct_filter)
+        LOG.debug(f'sacct args: {sacct_filter}')
+        db = sqlite3.connect(':memory:')
+        errors = slurm2sql(db, sacct_filter=sacct_filter,
+                           csv_input=getattr(args, 'csv_input', False) or csv_input)
+    return db
+
 
 def update_last_timestamp(db, update_time=None):
     """Update the last update time in the database, for resuming.
@@ -1011,7 +1091,8 @@ def compact_table():
         )
 
 
-SACCT_DEFAULT_FIELDS = 'JobID,User,State,Start,End,Partition,ExitCodeRaw,NodeList,NCPUS,CPUtime,CPUEff,ReqMem,MaxRSS,ReqGPUS,GPUUtil,TotDiskRead,TotDiskWrite,ReqTRES,AllocTRES,TRESUsageInTot,TRESUsageOutTot'
+SACCT_DEFAULT_FIELDS = "JobID,User,State,datetime(Start, 'unixepoch') AS Start,datetime(End, 'unixepoch') AS End,Partition,ExitCodeRaw,NodeList,NCPUS,CPUtime,CPUEff,AllocMem,TotalMem,MemEff,ReqGPUS,GPUEff,TotDiskRead,TotDiskWrite,ReqTRES,AllocTRES,TRESUsageInTot,TRESUsageOutTot"
+SACCT_DEFAULT_FIELDS_LONG = "JobID,User,State,datetime(Start, 'unixepoch') AS Start,datetime(End, 'unixepoch') AS End,Elapsed,Partition,ExitCodeRaw,NodeList,NCPUS,CPUtime,CPUEff,AllocMem,TotalMem,MemEff,ReqMem,MaxRSS,ReqGPUS,GPUEff,GPUUtil,TotDiskRead,TotDiskWrite,ReqTRES,AllocTRES,TRESUsageInTot,TRESUsageOutTot"
 COMPLETED_STATES = 'CA,CD,DL,F,NF,OOM,PR,RV,TO'
 def sacct_cli(argv=sys.argv[1:], csv_input=None):
     """A command line that uses slurm2sql to give an sacct-like interface."""
@@ -1026,13 +1107,11 @@ def sacct_cli(argv=sys.argv[1:], csv_input=None):
     parser.add_argument('--db',
                         help="Read from this DB.  Don't import new data.")
     parser.add_argument('--output', '-o', default=SACCT_DEFAULT_FIELDS,
-                        help="Fields to output (comma separated list, use '*' for all fields).  NOT safe from SQL injection")
+                        help="Fields to output (comma separated list, use '*' for all fields).  NOT safe from SQL injection.  If 'long' then some longer default list")
     parser.add_argument('--format', '-f', default=compact_table(),
                         help="Output format (see tabulate formats: https://pypi.org/project/tabulate/ (default simple)")
     parser.add_argument('--order',
                         help="SQL order by (arbitrary SQL expression using column names).  NOT safe from SQL injection.")
-    parser.add_argument('--completed', '-c', action='store_true',
-                        help=f"Select for completed job states ({COMPLETED_STATES})  You need to specify --starttime (-S) at some point in the past, due to how saccont default works (for example '-S now-1week').  This option automatically sets '-E now'")
     parser.add_argument('--csv-input',
                         help="Don't parse sacct but import this CSV file.  It's read with "
                              "Python's default csv reader (excel format).  Beware badly "
@@ -1041,6 +1120,16 @@ def sacct_cli(argv=sys.argv[1:], csv_input=None):
                         help="Don't output anything unless errors")
     parser.add_argument('--verbose', '-v', action='store_true',
                         help="Output more logging info")
+    # No --db compatibility
+    group = parser.add_argument_group(description="Selectors that only works when getting new data (not with --db):")
+    group.add_argument('--completed', '-c', action='store_true',
+                        help=f"Select for completed job states ({COMPLETED_STATES})  You need to specify --starttime (-S) at some point in the past, due to how saccont default works (for example '-S now-1week').  This option automatically sets '-E now'.  Not compatible with --db.")
+    group.add_argument('--running-at-time', metavar='TIME', help="Only jobs running at this time.  Not compatible with --db.  Expanded to --start=TIME --end=TIME --state=R.")
+    # --db compatibility
+    group = parser.add_argument_group(description="Selectors that also work with --db:")
+    group.add_argument('--user', '-u', help="Limit to this or these users.  Compatible with --db.")
+    group.add_argument('--partition', '-r', help="Jobs in this partition.  Works with --db.  Getting fresh data, an exact match and can be a comma separated list.  With --db, a raw glob match.")
+
     args, sacct_filter = parser.parse_known_args(argv)
 
     if args.verbose:
@@ -1048,20 +1137,17 @@ def sacct_cli(argv=sys.argv[1:], csv_input=None):
     if args.quiet:
         logging.lastResort.setLevel(logging.WARN)
     LOG.debug(args)
+    if args.output == 'long':
+        args.output = SACCT_DEFAULT_FIELDS_LONG
 
-    sacct_filter = process_sacct_filter(args, sacct_filter)
+    db = import_or_open_db(args, sacct_filter, csv_input=csv_input)
 
-    LOG.debug(f'sacct args: {sacct_filter}')
-    if args.db:
-        db = sqlite3.connect(args.db)
-    else:
-         db = sqlite3.connect(':memory:')
-         errors = slurm2sql(db, sacct_filter=sacct_filter,
-                            csv_input=args.csv_input or csv_input)
+    # If we run sacct, then args.user is set to None so we don't do double filtering here
+    where = args_to_sql_where(args)
 
     from tabulate import tabulate
-
-    cur = db.execute(f'select {args.output} from slurm')
+    cur = db.execute(f'select {args.output} from slurm WHERE true {where}',
+                     {'user':args.user, 'partition': args.partition})
     headers = [ x[0] for x in cur.description ]
     print(tabulate(cur, headers=headers, tablefmt=args.format))
 
@@ -1079,8 +1165,6 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
         jobs, use "--completed -S now-1week" (a start time must be
         given with --completed because of how sacct works).
 
-        MemReqGiB is amount requested per node (to compare with MaxRSSGiB).
-
         This only queries jobs with an End time (unlike most other commands).
 
         If a single argument is given, and it
@@ -1097,8 +1181,6 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                         help="Aggregate data by user.")
     parser.add_argument('--order',
                         help="SQL order by (arbitrary SQL expression using column names).  NOT safe from SQL injection.")
-    parser.add_argument('--completed', '-c', action='store_true',
-                        help=f"Select for completed job states ({COMPLETED_STATES})  You need to specify --starttime (-S) at some point in the past, due to how saccont default works (for example '-S now-1week').  This option automatically sets '-E now'.")
     parser.add_argument('--csv-input',
                         help="Don't parse sacct but import this CSV file.  It's read with "
                              "Python's default csv reader (excel format).  Beware badly "
@@ -1107,6 +1189,16 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                         help="Don't output anything unless errors")
     parser.add_argument('--verbose', '-v', action='store_true',
                         help="Output more logging info")
+    # No --db compatibility
+    group = parser.add_argument_group(description="Selectors that only works when getting new data (not with --db):")
+    group.add_argument('--completed', '-c', action='store_true',
+                        help=f"Select for completed job states ({COMPLETED_STATES})  You need to specify --starttime (-S) at some point in the past, due to how saccont default works (for example '-S now-1week').  This option automatically sets '-E now'.  Not compatible with --db.")
+    group.add_argument('--running-at-time', metavar='TIME', help="Only jobs running at this time.  Not compatible with --db.  Expanded to --start=TIME --end=TIME --state=R.")
+    # --db compatibility
+    group = parser.add_argument_group(description="Selectors that also work with --db:")
+    group.add_argument('--user', '-u', help="Limit to this or these users.  Compatible with --db.")
+    group.add_argument('--partition', '-r', help="Jobs in this partition.  Works with --db.  Getting fresh data, an exact match and can be a comma separated list.  With --db, a raw glob match.")
+
     args, sacct_filter = parser.parse_known_args(argv)
 
     if args.verbose:
@@ -1115,20 +1207,15 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
         logging.lastResort.setLevel(logging.WARN)
     LOG.debug(args)
 
-    sacct_filter = process_sacct_filter(args, sacct_filter)
-
     if args.order:
         order_by = f'ORDER BY {args.order}'
     else:
         order_by = ''
 
-    LOG.debug(f'sacct args: {sacct_filter}')
-    if args.db:
-        db = sqlite3.connect(args.db)
-    else:
-         db = sqlite3.connect(':memory:')
-         errors = slurm2sql(db, sacct_filter=sacct_filter,
-                            csv_input=args.csv_input or csv_input)
+    db = import_or_open_db(args, sacct_filter, csv_input=csv_input)
+
+    # If we run sacct, then args.user is set to None so we don't do double filtering here
+    where = args_to_sql_where(args)
 
     from tabulate import tabulate
 
@@ -1140,8 +1227,8 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                                 round(sum(Elapsed*NCPUS)/86400,1) AS cpu_day,
                                 printf("%2.0f%%", 100*sum(Elapsed*NCPUS*CPUEff)/sum(Elapsed*NCPUS)) AS CPUEff,
 
-                                round(sum(Elapsed*MemReq)/1073741824/86400,1) AS mem_GiB_day,
-                                printf("%2.0f%%", 100*sum(Elapsed*MemReq*MemEff)/sum(Elapsed*MemReq)) AS MemEff,
+                                round(sum(Elapsed*AllocMem)/1073741824/86400,1) AS mem_GiB_day,
+                                printf("%2.0f%%", 100*sum(Elapsed*AllocMem*MemEff)/sum(Elapsed*AllocMem)) AS MemEff,
 
                                 round(sum(Elapsed*NGPUs)/86400,1) AS gpu_day,
                                 iif(sum(NGpus), printf("%2.0f%%", 100*sum(Elapsed*NGPUs*GPUeff)/sum(Elapsed*NGPUs)), NULL) AS GPUEff,
@@ -1150,9 +1237,9 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                                 round(sum(TotDiskWrite/1048576)/sum(Elapsed),2) AS write_MiBps
 
                                 FROM eff
-                                WHERE End IS NOT NULL
+                                WHERE End IS NOT NULL {where}
                             GROUP BY user ) {order_by}
-                            """)
+                            """, {'user': args.user, 'partition': args.partition})
         headers = [ x[0] for x in cur.description ]
         data = cur.fetchall()
         if len(data) == 0:
@@ -1169,8 +1256,8 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                          NCPUS,
                          printf("%3.0f%%",round(CPUeff, 2)*100) AS "CPUeff",
 
-                         round(MemReq/1073741824,2) AS MemReqGiB,
-                         round(MaxRSS/1073741824,2) AS MaxRSSGiB,
+                         round(AllocMem/1073741824,2) AS MemAllocGiB,
+                         round(TotalMem/1073741824,2) AS MemTotGiB,
                          printf("%3.0f%%",round(MemEff,2)*100)  AS MemEff,
 
                          NGpus,
@@ -1181,7 +1268,7 @@ def seff_cli(argv=sys.argv[1:], csv_input=None):
                          round(TotDiskWrite/Elapsed/1048576,2) AS write_MiBps
 
                          FROM eff
-                         WHERE End IS NOT NULL ) {order_by}""")
+                         WHERE End IS NOT NULL {where} ) {order_by}""", {'user': args.user, 'partition': args.partition})
     headers = [ x[0] for x in cur.description ]
     data = cur.fetchall()
     if len(data) == 0:

{slurm2sql-0.9.2 → slurm2sql-0.9.4}/test.py

@@ -164,12 +164,12 @@ def test_cpueff(db):
 
 def test_gpueff(db):
     data = """
-    JobID,TRESUsageInAve
-    1,gres/gpuutil=23
+    JobID,AllocTRES,TRESUsageInTot
+    1,gres/gpu=1,gres/gpuutil=23
     """
     slurm2sql.slurm2sql(db, [], csv_input=csvdata(data))
     print(db.execute('select * from eff;').fetchall())
-    assert fetch(db, 1, 'GPUEff', table='eff') == 0.23
+    assert fetch(db, 1, 'GpuEff', table='eff') == 0.23
 
 
 #
@@ -230,18 +230,19 @@ def test_seff(db, capsys):
 
 def test_seff_mem(db, capsys):
     data = """
-    JobID,End,NNodes,NCPUS,ReqMem,MaxRSS
-    111,1970-01-01T00:00:00,1,1,10G,
-    111.2,,1,1,,8G
+    JobID,End,NNodes,NCPUS,ReqMem,MaxRSS,AllocTRES,TRESUsageInTot
+    111,1970-01-01T00:00:00,1,1,10G,,mem=10G,
+    111.2,,1,1,,8G,mem=10G,mem=6G
     """
+    # Changed 2025-04-23: no longer uses ReqMe.m and MaxRSS but AllocTRES and TRESUsageInTot
     slurm2sql.seff_cli(argv=[], csv_input=csvdata(data))
     captured = capsys.readouterr()
     assert '111' in captured.out
-    assert '80%' in captured.out
+    assert '60%' in captured.out
 
 def test_seff_gpu(db, capsys):
     data = """
-    JobID,End,Elapsed,TotalCPU,NCPUS,AllocTRES,TRESUsageInAve
+    JobID,End,Elapsed,TotalCPU,NCPUS,AllocTRES,TRESUsageInTot
     111,1970-01-01T00:00:00,,1,1,,
     111.2,1970-01-01T00:00:00,100,1,1,gres/gpu=1,gres/gpuutil=23
     """