qdir 1.0.0rc1__tar.gz

qdir-1.0.0rc1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Carl Osterwisch
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

qdir-1.0.0rc1/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: qdir
+Version: 1.0.0rc1
+Summary: A very low overhead first-in, first-out job queue for the local machine
+Author-email: Carl Osterwisch <costerwi@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/costerwi/qdir
+Project-URL: Issues, https://github.com/costerwi/qdir/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: filelock
+Requires-Dist: pywin32; sys_platform == "win32"
+Dynamic: license-file
+
+# qdir
+A very low overhead first-in, first-out job queue for the local machine
+
+The queue is maintained as a directory of tiny job queue files.
+When a new job is submitted, this script will identify any existing job queue files and then add its own queue file to the directory.
+No job will begin until all the preceeding queue files are gone.
+Each job will delete its own queue file upon completion so the next job can begin.
+A shared queue directory can be specified on a multi-user system.
+On Linux this is `/tmp` by default.
+
+Any command may be submitted to the job queue using qsub (see usage below).
+You may check the status of the queue (qstat) and remove jobs (qdel).
+
+# Installation
+Requires python3 running on Windows or Linux.
+One way to get python3 on Windows is through the [Microsoft Store](https://apps.microsoft.com/detail/9PNRBTZXMB4Z).
+Here is the command to install qdir once you have python3 available:
+```bash
+python3 -m pip install qdir
+```
+
+See [here](https://pip.pypa.io/en/stable/installation/) if there is "no module named pip."
+
+## Abaqus Finite Element Analysis (FEA) extras
+The package includes additional files to help Abaqus users submit and monitor the qdir job queue.
+Use the following command to configure Abaqus after qdir itself has been installed:
+```bash
+python3 -m qdir.abaqus install
+```
+
+By default, the necessary settings are appended to the end of your user `abaqus_v6.env` file.
+To uninstall you may removed the settings manually or with this command:
+```bash
+python3 -m qdir.abaqus uninstall
+```
+
+# Usage
+
+## Command line job submission
+Submit commands or scripts to the queue using the `qsub` command.
+For example, on Linux:
+```bash
+qsub ./myJobFile.sh
+```
+or on Windows:
+```bash
+qsub myJobFile.cmd
+```
+
+Any printed output from the command will be saved to a file based on the job name with the job Id appended to the end.
+
+Display command options with `qsub -h`
+
+## Command line queue status
+Check status with the `qstat` command, for example:
+```bash
+qstat
+```
+or, for full details:
+```bash
+qstat -f
+```
+
+## Command line job removal
+First use the `qstat` command above to find the jobIds and job names in the queue.
+Delete or kill jobs with the `qdel` command followed by a jobId or job name.
+For example, to delete job 12343:
+```bash
+qdel 12343
+```
+
+## Abaqus extras
+Submit Abaqus jobs as usual from the command line:
+```bash
+abaqus job=whatever
+```
+This will invoke the `qdir_submit.py` module to create a job script and submit it to qdir.
+
+You may submit jobs within Abaqus CAE by selecting `qdir` as the Job's queue before submitting.
+You may check the queue status within CAE in the menu `Plug-ins > QDir Job Queue Monitor...`
+Click on a Job to see more detailed information.
+Right-click on a Job to expose the menu allowing you to delete the Job.
+
+## Customization
+Most parameters used by the script may be customized with command line options, system environment variables, or directives within the job file itself.
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| -N *NAME*, --name *NAME*  | Name for the job. | The last argument of the command |
+| -q *QUEUE*, --queue *QUEUE* | Directory to store the queue files. | environment variable `QDIR_QUEUE` or `/tmp` on Linux |
+| -c *INTERVAL*, --interval *INTERVAL* | Polling interval in seconds. | environment variable `QDIR_POLL_INTERVAL` or 5 |
+| -i *INFO*, --info *INFO*  | Specify the format of information about jobs in queue | environment variable `QDIR_INFO` or `{jobId:8} {owner:12} {status:35} {name}` |
+| -o *OUTPUT*, --output *OUTPUT* | File name for combined stdout and stderr. | environment variable `QDIR_LOGNAME` or `{name}._{jobId}`
+| -s *STATUS*, --status *STATUS* | Specify the format of job status messages output to terminal. | environment variable `QDIR_STATUS` or `{jobId} {name} {status} after {duration}`
+| -v *VERBOSE*, --verbose *VERBOSE* | Set the verbosity level. | environment variable `QDIR_VERBOSE` or 2 |

qdir-1.0.0rc1/README.md

@@ -0,0 +1,95 @@
+# qdir
+A very low overhead first-in, first-out job queue for the local machine
+
+The queue is maintained as a directory of tiny job queue files.
+When a new job is submitted, this script will identify any existing job queue files and then add its own queue file to the directory.
+No job will begin until all the preceeding queue files are gone.
+Each job will delete its own queue file upon completion so the next job can begin.
+A shared queue directory can be specified on a multi-user system.
+On Linux this is `/tmp` by default.
+
+Any command may be submitted to the job queue using qsub (see usage below).
+You may check the status of the queue (qstat) and remove jobs (qdel).
+
+# Installation
+Requires python3 running on Windows or Linux.
+One way to get python3 on Windows is through the [Microsoft Store](https://apps.microsoft.com/detail/9PNRBTZXMB4Z).
+Here is the command to install qdir once you have python3 available:
+```bash
+python3 -m pip install qdir
+```
+
+See [here](https://pip.pypa.io/en/stable/installation/) if there is "no module named pip."
+
+## Abaqus Finite Element Analysis (FEA) extras
+The package includes additional files to help Abaqus users submit and monitor the qdir job queue.
+Use the following command to configure Abaqus after qdir itself has been installed:
+```bash
+python3 -m qdir.abaqus install
+```
+
+By default, the necessary settings are appended to the end of your user `abaqus_v6.env` file.
+To uninstall you may removed the settings manually or with this command:
+```bash
+python3 -m qdir.abaqus uninstall
+```
+
+# Usage
+
+## Command line job submission
+Submit commands or scripts to the queue using the `qsub` command.
+For example, on Linux:
+```bash
+qsub ./myJobFile.sh
+```
+or on Windows:
+```bash
+qsub myJobFile.cmd
+```
+
+Any printed output from the command will be saved to a file based on the job name with the job Id appended to the end.
+
+Display command options with `qsub -h`
+
+## Command line queue status
+Check status with the `qstat` command, for example:
+```bash
+qstat
+```
+or, for full details:
+```bash
+qstat -f
+```
+
+## Command line job removal
+First use the `qstat` command above to find the jobIds and job names in the queue.
+Delete or kill jobs with the `qdel` command followed by a jobId or job name.
+For example, to delete job 12343:
+```bash
+qdel 12343
+```
+
+## Abaqus extras
+Submit Abaqus jobs as usual from the command line:
+```bash
+abaqus job=whatever
+```
+This will invoke the `qdir_submit.py` module to create a job script and submit it to qdir.
+
+You may submit jobs within Abaqus CAE by selecting `qdir` as the Job's queue before submitting.
+You may check the queue status within CAE in the menu `Plug-ins > QDir Job Queue Monitor...`
+Click on a Job to see more detailed information.
+Right-click on a Job to expose the menu allowing you to delete the Job.
+
+## Customization
+Most parameters used by the script may be customized with command line options, system environment variables, or directives within the job file itself.
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| -N *NAME*, --name *NAME*  | Name for the job. | The last argument of the command |
+| -q *QUEUE*, --queue *QUEUE* | Directory to store the queue files. | environment variable `QDIR_QUEUE` or `/tmp` on Linux |
+| -c *INTERVAL*, --interval *INTERVAL* | Polling interval in seconds. | environment variable `QDIR_POLL_INTERVAL` or 5 |
+| -i *INFO*, --info *INFO*  | Specify the format of information about jobs in queue | environment variable `QDIR_INFO` or `{jobId:8} {owner:12} {status:35} {name}` |
+| -o *OUTPUT*, --output *OUTPUT* | File name for combined stdout and stderr. | environment variable `QDIR_LOGNAME` or `{name}._{jobId}`
+| -s *STATUS*, --status *STATUS* | Specify the format of job status messages output to terminal. | environment variable `QDIR_STATUS` or `{jobId} {name} {status} after {duration}`
+| -v *VERBOSE*, --verbose *VERBOSE* | Set the verbosity level. | environment variable `QDIR_VERBOSE` or 2 |

qdir-1.0.0rc1/pyproject.toml

@@ -0,0 +1,35 @@
+[project]
+name = "qdir"
+authors = [
+    { name="Carl Osterwisch", email="costerwi@gmail.com" },
+]
+description = "A very low overhead first-in, first-out job queue for the local machine"
+readme = "README.md"
+requires-python = ">=3.6"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+]
+license = "MIT"
+license-files = ["LICENSE"]
+dependencies = [
+    "filelock",
+    "pywin32; sys_platform == 'win32'",
+]
+dynamic = ["version"]
+
+[project.scripts]
+qdel = "qdir.qdel:main"
+qstat = "qdir.qstat:main"
+qsub = "qdir.qsub:main"
+
+[project.urls]
+Homepage = "https://github.com/costerwi/qdir"
+Issues = "https://github.com/costerwi/qdir/issues"
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.dynamic]
+version = {attr = "qdir.__version__"}

qdir-1.0.0rc1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

qdir-1.0.0rc1/src/qdir/__init__.py

@@ -0,0 +1,275 @@
+#!/bin/env python3
+"""Implementation of a first-in, first-out job queue for the local machine.
+
+The queue is maintained as a directory of tiny job queue files.
+When a new job is submitted, this script will identify any existing job queue files and then add its own queue file to the directory.
+No job will begin until all the preceeding queue files are gone.
+Each job will delete its own queue file upon completion so the next job can begin.
+A shared queue directory should be specified on a multi-user system.
+"""
+
+__author__ = "Carl Osterwisch"
+__version__ = "1.0.0rc1"
+__url__ = "https://github.com/costerwi/qdir/"
+
+import os
+import signal
+import subprocess
+import sys
+from argparse import ArgumentParser, Namespace
+from pathlib import Path
+from tempfile import gettempdir
+from time import asctime, perf_counter, sleep
+
+from filelock import FileLock
+
+default = Namespace
+
+parser = ArgumentParser(epilog=f"qdir {__version__} {__url__}")
+default.queue = Path(os.environ.get("QDIR_QUEUE", gettempdir()))
+parser.add_argument(
+    "-q",
+    "--queue",
+    type=Path,
+    help="Directory to store the queue files. "
+    f"Default is environment variable QDIR_QUEUE or {default.queue}",
+    default=default.queue,
+)
+default.verbose = int(os.environ.get("QDIR_VERBOSE", "2"))
+parser.add_argument(
+    "-v",
+    "--verbose",
+    type=int,
+    help="Verbosity level. "
+    f"Default is environment variable QDIR_VERBOSE or {default.verbose}",
+    default=default.verbose,
+)
+
+qFileSuffix = os.environ.get("QDIR_SUFFIX", ".qdir")  # suffix for queue files
+
+
+class Queue:
+    def __init__(self, **kwargs):
+        """Create the queue, allowing parameters to override defaults"""
+        for parameter, value in vars(default).items():
+            setattr(self, parameter, kwargs.pop(parameter, value))
+        if not self.queue.is_dir():
+            raise FileNotFoundError(f"The QDIR directory {self.queue} does not exist")
+        if not os.access(self.queue, os.W_OK):
+            raise PermissionError(f"The QDIR directory {self.queue} is not writable")
+
+    def getQueuedJobs(self) -> list[Path]:
+        """Return a list of qFilePath for currently queued jobs, sorted oldest to newest."""
+        # Collecct a set of active process ids
+        if os.name == "nt":
+            import win32process
+
+            processIds = {str(pid) for pid in win32process.EnumProcesses()}
+        else:
+            procPath = Path("/proc")
+            processIds = {p.stem for p in procPath.iterdir() if p.stem.isnumeric()}
+
+        def is_qFile(path):
+            """Verify the path is an active job file with a matching process id"""
+            if not path.stem.isnumeric():
+                return False  # file name must be an integer
+            if not (path.is_file()):
+                return False  # path must represent a file
+            if path.stem in processIds:
+                return True  # file name must match a running process number
+
+            # Otherwise, the process may have died or the computer was rebooted with
+            # this job in the queue. This orphaned qFile may be removed.
+            print(
+                "Removing orphaned queue file",
+                self.info.format(**self.getJobInfo(path)),
+                file=sys.stderr,
+            )
+            try:
+                path.unlink(missing_ok=True)
+            except OSError as e:
+                print(e, file=sys.stderr)
+
+        queuedJobs = [
+            filePath
+            for filePath in self.queue.glob("*" + qFileSuffix)
+            if is_qFile(filePath)
+        ]
+        return sorted(queuedJobs, key=lambda qFilePath: qFilePath.stat().st_mtime)
+
+    def getJobInfo(self, qFilePath: Path) -> dict:
+        """Return information about the job specified in qFilePath"""
+        jobInfo = {
+            "jobId": qFilePath.stem,
+            "owner": qFilePath.owner(),
+            "name": "",
+            "command": "",
+            "directory": "",
+            "behind": "",
+            "status": "",
+            "history": [],  # list[str]
+        }
+        try:
+            with qFilePath.open() as qFile:
+                for line in qFile:
+                    keyValue = line.strip().split("\t", 2)
+                    if len(keyValue) == 2:
+                        jobInfo.update([keyValue])
+                    else:
+                        status = keyValue[0]
+                        jobInfo["history"].append(status)
+                        jobInfo["status"] = status  # latest update
+        except OSError as e:
+            status = repr(e)
+            jobInfo["history"].append(status)
+            jobInfo["status"] = status
+        return jobInfo
+
+    def dequeue(self, qFilePath: Path):
+        """Remove a job from the queue"""
+        assert qFilePath.parent == self.queue
+        assert qFilePath.suffix == qFileSuffix
+        assert qFilePath.stem.isnumeric()
+        # Removing qFile will dequeue the job but the effect is delayed by polling interval
+        qFilePath.unlink(missing_ok=True)
+        if os.name == "posix":
+            # On Linux the SIGTERM signal has immediate effect
+            try:
+                os.kill(int(qFilePath.stem), signal.SIGTERM)
+            except OSError:  # no such process
+                pass
+
+    def enqueue(self, cmd: list[str]):
+        """Wait in queue until prior jobs finish, then run cmd"""
+        import getpass
+        import shutil
+
+        if len(cmd) < 1:
+            raise SyntaxError("Missing cmd")
+        if self.verbose >= 1 and not shutil.which(cmd[0]):
+            print(f"WARNING: {cmd[0]} is not in PATH", file=sys.stderr)
+        if not self.interval > 0:
+            self.interval = 1
+
+        startTime = perf_counter()  # time job was enqueued
+        job = None  # subprocess running the cmd, if any
+
+        jobInfo = {  # information about this job
+            "jobId": str(os.getpid()),  # process id of this script
+            "owner": getpass.getuser(),
+            "name": self.name or cmd[-1],
+            "command": " ".join(cmd),
+            "directory": Path.cwd(),
+        }
+
+        def updateStatus(priority, status, file=None):
+            """Notify user of status change and optionally record in file"""
+            jobInfo["status"] = status
+            jobInfo["time"] = asctime()
+            jobInfo["duration"] = humanDuration(perf_counter() - startTime)
+            if self.verbose >= priority:
+                # Echo status to the terminal
+                print(self.status.format(**jobInfo))
+            if file is not None:
+                # Record status history in specified qFile
+                print("{time} {status}".format(**jobInfo), file=file)
+
+        qFilePath = (self.queue / jobInfo["jobId"]).with_suffix(qFileSuffix)
+        lock = FileLock(self.queue / "qdir.lock")  # prevent previousJobs race
+        with lock.acquire(timeout=5):
+            previousJobs = self.getQueuedJobs()
+            jobInfo["behind"] = " ".join([job.stem for job in previousJobs])
+            with qFilePath.open("w") as qFile:
+                os.environ["QDIR_FILE"] = qFile.name  # will be exported to cmd
+                # Initialize qFile contents
+                for key in "name", "command", "directory", "behind":
+                    value = jobInfo.get(key)
+                    if value:
+                        print(key, value, sep="\t", file=qFile)
+                updateStatus(priority=2, status="Queued", file=qFile)
+
+        def terminateJob(sig=None, frame=None):
+            """Terminate this job if running or dequeue if not"""
+            nonlocal job
+            result = ""
+            qFilePath.unlink(missing_ok=True)  # remove its qFile
+            if job is None:
+                # removed from the queue before starting
+                result = ".. Dequeued"
+            elif hasattr(job, "pid"):  # job is or was running
+                if job.poll() is None:
+                    # job is still running
+                    if self.verbose >= 2:
+                        result = ".. Killed"
+                    if os.name == "nt":
+                        job.send_signal(signal.CTRL_BREAK_EVENT)
+                    else:
+                        # pass SIGTERM signal to all child processes in this process group
+                        os.killpg(os.getpgid(job.pid), signal.SIGTERM)
+                elif self.verbose >= 1:
+                    # job is no longer running; check returncode
+                    if job.returncode <= 0:
+                        result = "** Done"
+                    else:
+                        result = "-- Error"
+            if result:
+                updateStatus(priority=2, status=result)
+                job = result  # prevent this from duplicate reporting
+
+        signal.signal(
+            signal.SIGTERM, terminateJob
+        )  # catch termination signal (ignored on Windows)
+
+        # Wait for all previousJobs to complete
+        previousJobs.reverse()  # check newest jobs first
+        while qFilePath.exists() and any(
+            otherFile.exists() for otherFile in previousJobs
+        ):
+            sleep(self.interval)
+
+        # Launch the cmd only if this job's qFile still exists
+        if qFilePath.exists():
+            try:
+                # Append the job starting status to qFile
+                stat = qFilePath.stat()
+                with qFilePath.open("a") as qFile:
+                    print(
+                        "Waited in queue",
+                        humanDuration(perf_counter() - startTime),
+                        file=qFile,
+                    )
+                    updateStatus(priority=3, status="Started", file=qFile)
+                # Keep mtime unchanged so the job files sort correctly
+                os.utime(qFilePath, (stat.st_atime, stat.st_mtime))
+            except OSError:
+                pass  # ignore status update exceptions
+            logFilePath = Path(self.output.format(**jobInfo))
+            with logFilePath.open("w") as logFile:
+                opts = {}
+                if os.name == "nt":
+                    opts["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP
+                try:
+                    job = subprocess.Popen(
+                        cmd,
+                        shell=False,
+                        start_new_session=True,
+                        stdin=subprocess.DEVNULL,
+                        stdout=logFile,
+                        stderr=subprocess.STDOUT,
+                        **opts,
+                    )
+                    startTime = perf_counter()  # reset to time job started running
+                    while qFilePath.exists() and job.poll() is None:
+                        # job is still running
+                        sleep(self.interval)
+                except OSError as e:
+                    print(e, file=logFile)
+
+        terminateJob()  # Clean up at the end
+
+
+def humanDuration(seconds: float) -> str:
+    for unit, per in ("days", 86400), ("hours", 3600), ("minutes", 60):
+        if seconds >= 1.1 * per:
+            return f"{seconds / per:.2g} {unit}"
+    return f"{seconds:.2g} seconds"

qdir-1.0.0rc1/src/qdir/abaqus/__init__.py

@@ -0,0 +1,5 @@
+"""Modules to help Abaqus interact with the qdir job queue
+
+Install with:
+    python3 -m qdir.abaqus install
+"""

qdir-1.0.0rc1/src/qdir/abaqus/__main__.py

@@ -0,0 +1,66 @@
+"""Install or uninstall qdir for Abaqus.
+
+This script will modify the user's abaqus_v6.env file with
+settings for the qdir job queue.
+"""
+
+from __future__ import print_function, with_statement  # use Python3 syntax
+
+import os
+import sys
+from argparse import ArgumentParser
+
+settings = """# Settings for qdir job queue
+sys.path.append(r"{}")  # directory of qdir_submit.py
+import qdir_submit
+qdir_submit.init(queues)
+plugin_central_dir += os.pathsep + qdir_submit.dirname
+run_mode = BATCH  # command line will submit to queue by default"""
+
+parser = ArgumentParser(description=__doc__)
+parser.add_argument(
+    "action",
+    choices=["install", "uninstall"],
+    help="Specify what to do.",
+)
+default = os.path.expanduser("~/abaqus_v6.env")
+parser.add_argument(
+    "env",
+    nargs="?",
+    default=default,
+    help="Optionally specify the .env file to modify. Default is {}".format(default),
+)
+
+
+def uninstall(user_env):
+    """Remove qdir settings from user's abaqus_v6.env"""
+    new_env = user_env + ".new"
+    with open(new_env, "w") as new_env_file:
+        try:
+            # copy current settings
+            with open(user_env) as env_file:
+                for line in env_file:
+                    if not "qdir" in line:  # skip any qdir settings
+                        print(line.rstrip(), file=new_env_file)
+            os.remove(user_env)
+        except FileNotFoundError:
+            pass
+    os.rename(new_env, user_env)
+
+
+def install(user_env):
+    """Append qdir settings to user's abaqus_v6.env"""
+    uninstall(user_env)  # remove any previous settings
+    dirname = os.path.dirname(os.path.normpath(__file__))
+    with open(user_env, "a") as user_env_file:
+        print(settings.format(dirname), file=user_env_file)
+
+
+if __name__ == "__main__":
+    options = parser.parse_args(sys.argv[1:])
+    if "uninstall" == options.action:
+        print("Removing qdir settings from", options.env)
+        uninstall(options.env)
+    else:
+        print("Adding qdir settings to", options.env)
+        install(options.env)