lsst-ctrl-bps-htcondor 30.0.0rc2__tar.gz → 30.0.1rc1__tar.gz

{lsst_ctrl_bps_htcondor-30.0.0rc2/python/lsst_ctrl_bps_htcondor.egg-info → lsst_ctrl_bps_htcondor-30.0.1rc1}/PKG-INFO

@@ -1,20 +1,20 @@
 Metadata-Version: 2.4
 Name: lsst-ctrl-bps-htcondor
-Version: 30.0.0rc2
+Version: 30.0.1rc1
 Summary: HTCondor plugin for lsst-ctrl-bps.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License-Expression: BSD-3-Clause OR GPL-3.0-or-later
 Project-URL: Homepage, https://github.com/lsst/ctrl_bps_htcondor
+Project-URL: Source, https://github.com/lsst/ctrl_bps_htcondor
 Keywords: lsst
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering :: Astronomy
-Requires-Python: >=3.11.0
+Requires-Python: >=3.12.0
 Description-Content-Type: text/x-rst
 License-File: COPYRIGHT
 License-File: LICENSE

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/doc/lsst.ctrl.bps.htcondor/CHANGES.rst

@@ -1,3 +1,45 @@
+lsst-ctrl-bps-htcondor v30.0.1 (2026-02-03)
+===========================================
+
+Dropped support for Python 3.11.
+Tested with Python 3.14.
+
+Other Changes and Additions
+---------------------------
+
+- - Switched docs action to use ``sphinxutils``.
+  - Bumped minimum Python version to 3.12.
+  - Dropped ``black`` and ``isort`` sections from ``pyproject.toml``. (`DM-54005 <https://rubinobs.atlassian.net/browse/DM-54005>`_)
+
+
+lsst-ctrl-bps-htcondor v30.0.0 (2026-01-16)
+===========================================
+
+New Features
+------------
+
+- Added ability to bring the output files produced during job's execution back to the submit site. (`DM-48479 <https://rubinobs.atlassian.net/browse/DM-48479>`_)
+- Added ability to save job stdout/stderr across job retries by setting ``overwriteJobFiles`` to ``false`` in the submit yaml. (`DM-51905 <https://rubinobs.atlassian.net/browse/DM-51905>`_)
+- Added a mechanism that lets the user configure DAGMan via submit YAML. (`DM-52778 <https://rubinobs.atlassian.net/browse/DM-52778>`_)
+- Added using the given priority as the DAG node priority in addition to the HTCondor job priority. (`DM-53753 <https://rubinobs.atlassian.net/browse/DM-53753>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed bug causing ``bps status --id`` to always report ``MISFIT`` status.  Bug was introduced in DM-51868. (`DM-53014 <https://rubinobs.atlassian.net/browse/DM-53014>`_)
+- Fixed bug causing finalJob's memory to not be increased on retries like on regular payload jobs. (`DM-53053 <https://rubinobs.atlassian.net/browse/DM-53053>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- Redirected payload's ``stderr`` and ``stdout`` to the same file (``*.out``) to reduce number of files the plugin creates during submission. (`DM-31879 <https://rubinobs.atlassian.net/browse/DM-31879>`_)
+- Added documentation on ``condor_tail`` and ``condor_ssh_to_job``. (`DM-50900 <https://rubinobs.atlassian.net/browse/DM-50900>`_)
+- Fixed issue with some failed jobs being reported as deleted when using ``bps report`` with HTCondor plugin. (`DM-51868 <https://rubinobs.atlassian.net/browse/DM-51868>`_)
+- Divided ``htcondor_service.py`` and ``test_htcondor_service.py`` into smaller files. (`DM-52552 <https://rubinobs.atlassian.net/browse/DM-52552>`_)
+
+
 lsst-ctrl-bps-htcondor v29.1.0 (2025-06-13)
 ===========================================
 
@@ -136,8 +178,8 @@ Other Changes and Additions
 - Make HTCondor treat all jobs exiting with a signal as if they ran out of memory. (`DM-32968 <https://rubinobs.atlassian.net/browse/DM-32968>`_)
 - Make HTCondor plugin pass a group and user attribute to any batch systems that require such attributes for accounting purposes. (`DM-33887 <https://rubinobs.atlassian.net/browse/DM-33887>`_)
 
-ctrl_bps v23.0.0 (2021-12-10)
-=============================
+lsst-ctrl-bps-htcondor v23.0.0 (2021-12-10)
+===========================================
 
 New Features
 ------------

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/doc/lsst.ctrl.bps.htcondor/userguide.rst

@@ -1,3 +1,6 @@
+User Guide
+==========
+
 .. _htc-plugin-overview:
 
 Overview
@@ -6,7 +9,7 @@ Overview
 LSST Batch Processing Service (BPS) allows large-scale workflows to execute in
 well-managed fashion, potentially in multiple environments.  The service is
 provided by the `ctrl_bps`_ package.  ``ctrl_bps_htcondor`` is a plugin
-allowing `ctrl_bps` to execute workflows on computational resources managed by
+allowing ``ctrl_bps`` to execute workflows on computational resources managed by
 `HTCondor`_.
 
 .. _htc-plugin-preqs:
@@ -89,8 +92,8 @@ Job Environment
 ^^^^^^^^^^^^^^^
 
 By default, the htcondor jobs copy the environment from the shell in which
-`bps submit` was executed.  To set or override an environment variable via
-submission yaml, use an `environment` section.  Other yaml values and pre-existing
+``bps submit`` was executed.  To set or override an environment variable via
+submission yaml, use an ``environment`` section.  Other yaml values and pre-existing
 environment variables can be used.  Some examples:
 
 .. code-block:: YAML
@@ -104,7 +107,7 @@ environment variables can be used.  Some examples:
 
 .. note::
 
-   The `environment` section has to be at the root level.  There is no
+   The ``environment`` section has to be at the root level.  There is no
    way to change the environment inside another level (e.g., per site,
    per cluster, per pipeline task)
 
@@ -397,11 +400,11 @@ by ``provisioningMaxIdleTime`` (default value: 15 min., see below) or maximum
 wall time is reached.
 
 The provisioning job is expected to run as long as the workflow.  If the job
-dies, the job status will be `FAILED`.  If the job just completed successfully,
-the job status will be `SUCCEEDED` with a message saying it ended early (which
+dies, the job status will be ``FAILED``.  If the job just completed successfully,
+the job status will be ``SUCCEEDED`` with a message saying it ended early (which
 may or may not cause a problem since existing glideins could remain running).
 To get more information about either of these cases, check the job output
-and error files in the `jobs/provisioningJob` subdirectory.
+and error files in the ``jobs/provisioningJob`` subdirectory.
 
 
 If the automatic provisioning of the resources is enabled, the script that the

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "lsst-ctrl-bps-htcondor"
-requires-python = ">=3.11.0"
+requires-python = ">=3.12.0"
 description = "HTCondor plugin for lsst-ctrl-bps."
 license = "BSD-3-Clause OR GPL-3.0-or-later"
 license-files = ["COPYRIGHT", "LICENSE", "bsd_license.txt", "gpl-v3.0.txt"]
@@ -16,7 +16,6 @@ classifiers = [
     "Intended Audience :: Science/Research",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: 3.14",
@@ -36,6 +35,7 @@ dynamic = ["version"]
 
 [project.urls]
 "Homepage" = "https://github.com/lsst/ctrl_bps_htcondor"
+"Source" = "https://github.com/lsst/ctrl_bps_htcondor"
 
 [project.optional-dependencies]
 test = [
@@ -93,15 +93,6 @@ version = { attr = "lsst_versions.get_lsst_version" }
         name = "An API Removal or Deprecation"
         showcontent = true
 
-[tool.black]
-line-length = 110
-target-version = ["py311"]
-
-[tool.isort]
-profile = "black"
-line_length = 110
-known_first_party = ["lsst"]
-
 [tool.lsst_versions]
 write_to = "python/lsst/ctrl/bps/htcondor/version.py"
 
@@ -127,7 +118,7 @@ exclude_lines = [
 ]
 
 [tool.ruff]
-target-version = "py311"
+target-version = "py312"
 line-length = 110
 exclude = [
     "__init__.py",
@@ -194,6 +185,7 @@ checks = [
     "RT01",  # Unfortunately our @property trigger this.
     "RT02",  # Does not want named return value. DM style says we do.
     "SS05",  # pydocstyle is better at finding infinitive verb.
+    "RT03",  # Temporarily disable for sphinx empty returns.
 ]
 exclude = [
     '^__init__$',

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/python/lsst/ctrl/bps/htcondor/dagman_configurator.py

@@ -91,7 +91,7 @@ class DagmanConfigurator:
     ----------
     config : `lsst.ctrl.bps.BpsConfig`
         BPS configuration.
-    search_opts : `dict` [`str`, `Any`], optional
+    search_opts : `dict` [`str`, `~typing.Any`], optional
         Options to use while searching the BPS configuration for values.
 
     Raises

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/python/lsst/ctrl/bps/htcondor/htcondor_service.py

@@ -100,7 +100,7 @@ class HTCondorService(BaseWmsService):
 
         Returns
         -------
-        workflow : `lsst.ctrl.bps.wms.htcondor.HTCondorWorkflow`
+        workflow : `lsst.ctrl.bps.htcondor.HTCondorWorkflow`
             HTCondor workflow ready to be run.
         """
         _LOG.debug("out_prefix = '%s'", out_prefix)
@@ -141,7 +141,7 @@ class HTCondorService(BaseWmsService):
 
         Parameters
         ----------
-        workflow : `lsst.ctrl.bps.BaseWorkflow`
+        workflow : `lsst.ctrl.bps.htcondor.HTCondorWorkflow`
             A single HTCondor workflow to submit.  run_id is updated after
             successful submission to WMS.
         **kwargs : `~typing.Any`

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/python/lsst/ctrl/bps/htcondor/lssthtc.py

@@ -234,6 +234,7 @@ HTC_VALID_JOB_DAG_KEYS = {
     "retry_unless_exit",
     "abort_dag_on",
     "abort_exit",
+    "priority",
 }
 HTC_VERSION = version.parse(htcondor.__version__)
 
@@ -243,7 +244,7 @@ class RestrictedDict(MutableMapping):
 
     Parameters
     ----------
-    valid_keys : `Container`
+    valid_keys : `~collections.abc.Container`
         Strings that are valid keys.
     init_data : `dict` or `RestrictedDict`, optional
         Initial data.
@@ -490,7 +491,7 @@ def htc_write_attribs(stream, attrs):
 
     Parameters
     ----------
-    stream : `~io.TextIOBase`
+    stream : `~typing.TextIO`
         Output text stream (typically an open file).
     attrs : `dict`
         HTCondor job attributes (dictionary of attribute key, value).
@@ -512,7 +513,7 @@ def htc_write_condor_file(
 
     Parameters
     ----------
-    filename : `str` or os.PathLike
+    filename : `str` or `os.PathLike`
         Filename for the HTCondor submit file.
     job_name : `str`
         Job name to use in submit file.
@@ -823,7 +824,7 @@ def _htc_write_job_commands(stream, name, commands):
 
     Parameters
     ----------
-    stream : `~io.TextIOBase`
+    stream : `~typing.TextIO`
         Writeable text stream (typically an opened file).
     name : `str`
         Job name.
@@ -884,6 +885,12 @@ def _htc_write_job_commands(stream, name, commands):
             file=stream,
         )
 
+    if "priority" in commands and commands["priority"]:
+        print(
+            f"PRIORITY {name} {commands['priority']}",
+            file=stream,
+        )
+
 
 class HTCJob:
     """HTCondor job for use in building DAG.
@@ -974,7 +981,7 @@ class HTCJob:
 
         Parameters
         ----------
-        stream : `IO` or `str`
+        stream : `~typing.TextIO`
             Output Stream.
         dag_rel_path : `str`
             Relative path of dag to submit directory.
@@ -1002,7 +1009,7 @@ class HTCJob:
 
         Parameters
         ----------
-        fh : `~io.TextIOBase`
+        fh : `~typing.TextIO`
             Output stream.
         """
         printer = pprint.PrettyPrinter(indent=4, stream=fh)
@@ -1016,8 +1023,9 @@ class HTCDag(networkx.DiGraph):
 
     Parameters
     ----------
-    data : networkx.DiGraph.data
-        Initial graph.
+    data : `~typing.Any`
+        Initial graph data of any format that is supported
+        by the to_network_graph() function.
     name : `str`
         Name for DAG.
     """
@@ -1209,7 +1217,7 @@ class HTCDag(networkx.DiGraph):
 
         Parameters
         ----------
-        fh : `io.IO` or `str`
+        fh : `typing.IO`
             Where to dump DAG info as text.
         """
         for key, value in self.graph:
@@ -1296,7 +1304,7 @@ def condor_query(constraint=None, schedds=None, query_func=htc_query_present, **
         HTCondor schedulers which to query for job information. If None
         (default), the query will be run against the history file of
         the local scheduler only.
-    query_func : callable
+    query_func : `~collections.abc.Callable`
         An query function which takes following arguments:
 
         - ``schedds``: Schedulers to query (`list` [`htcondor.Schedd`]).
@@ -1653,7 +1661,7 @@ def read_dag_status(wms_path: str | os.PathLike) -> dict[str, Any]:
 
     Parameters
     ----------
-    wms_path : `str` or `os.PathLike
+    wms_path : `str` or `os.PathLike`
         Path that includes node status file for a run.
 
     Returns

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/python/lsst/ctrl/bps/htcondor/prepare_utils.py

@@ -306,7 +306,11 @@ def _translate_dag_cmds(gwjob):
         DAGMan commands for the job.
     """
     # Values in the dag script that just are name mappings.
-    dag_translation = {"abort_on_value": "abort_dag_on", "abort_return_value": "abort_exit"}
+    dag_translation = {
+        "abort_on_value": "abort_dag_on",
+        "abort_return_value": "abort_exit",
+        "priority": "priority",
+    }
 
     dagcmds = {}
     for gwkey, htckey in dag_translation.items():

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/python/lsst/ctrl/bps/htcondor/version.py

@@ -1,2 +1,2 @@
 __all__ = ["__version__"]
-__version__ = "30.0.0rc2"
+__version__ = "30.0.1rc1"

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1/python/lsst_ctrl_bps_htcondor.egg-info}/PKG-INFO

@@ -1,20 +1,20 @@
 Metadata-Version: 2.4
 Name: lsst-ctrl-bps-htcondor
-Version: 30.0.0rc2
+Version: 30.0.1rc1
 Summary: HTCondor plugin for lsst-ctrl-bps.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License-Expression: BSD-3-Clause OR GPL-3.0-or-later
 Project-URL: Homepage, https://github.com/lsst/ctrl_bps_htcondor
+Project-URL: Source, https://github.com/lsst/ctrl_bps_htcondor
 Keywords: lsst
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering :: Astronomy
-Requires-Python: >=3.11.0
+Requires-Python: >=3.12.0
 Description-Content-Type: text/x-rst
 License-File: COPYRIGHT
 License-File: LICENSE

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/tests/test_lssthtc.py

@@ -724,6 +724,7 @@ class HtcWriteJobCommands(unittest.TestCase):
             "retry": 3,
             "retry_unless_exit": 1,
             "abort_dag_on": {"node_exit": 100, "abort_exit": 4},
+            "priority": 123,
         }
 
         truth = """SCRIPT DEFER 1 120 DEBUG debug_pre.txt ALL PRE job1 exec1 arg1 arg2
@@ -733,6 +734,7 @@ VARS job1 spaces="a space"
 PRE_SKIP job1 1
 RETRY job1 3 UNLESS-EXIT 1
 ABORT-DAG-ON job1 100 RETURN 4
+PRIORITY job1 123
 """
         mockfh = io.StringIO()
         lssthtc._htc_write_job_commands(mockfh, "job1", dag_cmds)

{lsst_ctrl_bps_htcondor-30.0.0rc2 → lsst_ctrl_bps_htcondor-30.0.1rc1}/tests/test_prepare_utils.py

@@ -134,6 +134,19 @@ class TranslateJobCmdsTestCase(unittest.TestCase):
         self.assertEqual(htc_commands["max_retries"], 0)
 
 
+class TranslateDagCmdsTestCase(unittest.TestCase):
+    """Test _translate_dag_cmds method."""
+
+    def setUp(self):
+        self.gw_exec = GenericWorkflowExec("test_exec", "/dummy/dir/pipetask")
+
+    def testPriority(self):
+        gwjob = GenericWorkflowJob("priority", "label1", executable=self.gw_exec)
+        gwjob.priority = 100
+        dag_commands = prepare_utils._translate_dag_cmds(gwjob)
+        self.assertEqual(dag_commands["priority"], 100)
+
+
 class GroupToSubdagTestCase(unittest.TestCase):
     """Test _group_to_subdag function."""