lsst-pipe-base 29.2025.1200__tar.gz → 29.2025.1400__tar.gz

{lsst_pipe_base-29.2025.1200/python/lsst_pipe_base.egg-info → lsst_pipe_base-29.2025.1400}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-pipe-base
-Version: 29.2025.1200
+Version: 29.2025.1400
 Summary: Pipeline infrastructure for the Rubin Science Pipelines.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License: BSD 3-Clause License

{lsst_pipe_base-29.2025.1200 → lsst_pipe_base-29.2025.1400}/doc/lsst.pipe.base/CHANGES.rst

@@ -1,3 +1,56 @@
+lsst-pipe-base v29.0.0 (2025-03-25)
+===================================
+
+New Features
+------------
+
+- * Modified ``QuantumContext`` such that it now tracks all datasets that are retrieved and records them in ``dataset_provenance``.
+    This provenance is then passed to Butler on ``put()``.
+  * Added ``QuantumContext.add_additional_provenance()`` to allow a pipeline task author to attach additional provenance information to be recorded and associated with a particular input dataset. (`DM-35396 <https://rubinobs.atlassian.net/browse/DM-35396>`_)
+- Finished support for "steps" in pipelines and pipeline graphs.
+
+  Steps are an ordered sequence of special pipeline subsets that provide extra information and validation about how pipelines should actually be run. (`DM-46023 <https://rubinobs.atlassian.net/browse/DM-46023>`_)
+- Added Mermaid for pipeline and quantum graph visualization with tools for documentation/presentation purposes. (`DM-46503 <https://rubinobs.atlassian.net/browse/DM-46503>`_)
+- Added ``zip-from-graph`` subcommand for ``butler`` command-line to enable output artifacts associated with a graph to be combined into a Zip archive. (`DM-46776 <https://rubinobs.atlassian.net/browse/DM-46776>`_)
+- Added ``UpstreamFailureNoWorkFound``, an exception that is handled the same way as ``NoWorkFound`` that indicates that the root problem is probably in an upstream ``PipelineTask``. (`DM-46948 <https://rubinobs.atlassian.net/browse/DM-46948>`_)
+- Plugin discovery is now automated through Python entry points when using ``pip``.
+  It is now an error if the ``DAF_BUTLER_PLUGINS`` environment variable is set for this package. (`DM-47143 <https://rubinobs.atlassian.net/browse/DM-47143>`_)
+- * Added new command-line ``butler retrieve-artifacts-for-quanta`` which can be used to retrieve input or output datasets associated with a graph or specific quanta.
+  * Added new ``QuantumGraph.get_refs()`` method to retrieve dataset refs from a graph. (`DM-47328 <https://rubinobs.atlassian.net/browse/DM-47328>`_)
+- Add the ``QuantumSuccessCaveats`` flag enum, which can be used to report on ``NoWorkFound`` and other qualified successes in execution.
+
+  This adds the flag enum itself and functionality in ``QuantumProvenanceGraph`` (which backs ``pipetask report --force-v2``) to include it in reports.
+  It relies on additional changes in ``lsst.ctrl.mpexec.SingleQuantumExecutor`` to write the caveat flags into task metadata. (`DM-47730 <https://rubinobs.atlassian.net/browse/DM-47730>`_)
+- ``QuantumProvenanceGraph`` and ``pipetask report --force-v2`` can now report on exceptions raised and then ignored via the ``--no-raise-on-partial-outputs`` option.
+
+  Exceptions that lead to task failures are not yet tracked, because we do not write task metadata for failures and hence have nowhere to put the information. (`DM-48536 <https://rubinobs.atlassian.net/browse/DM-48536>`_)
+- Swapped to the new butler query system in ``QuantumGraph`` generation.
+
+  This change should be mostly transparent to users, aside from small changes in speed (typically faster, but not always). (`DM-45896 <https://rubinobs.atlassian.net/browse/DM-45896>`)
+
+Bug Fixes
+---------
+
+- Fixed a bug in QG generation that could lead to an error when using ``--skip-existing-in`` on a collection that had a successful quantum that did not write all of its predicted outputs. (`DM-49266 <https://rubinobs.atlassian.net/browse/DM-49266>`_)
+
+
+Performance Enhancement
+-----------------------
+
+- ``QuantumGraph`` generation has been partially rewritten to support building larger graphs and build all graphs faster.
+
+  With this change, ``QuantumGraph`` generation no longer uses a long-lived temporary table in the butler database for followup queries, and instead uploads a set of data IDs to the database for each query.
+  In addition, the algorithm for adding nodes and edges from the data ID query results has been reworked to duplicate irrelevant dimensions earlier, making it much faster. (`DM-49296 <https://rubinobs.atlassian.net/browse/DM-49296>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- Modified ``TaskMetadata`` such that it can now be assigned an empty list.
+  This list can be retrieved with ``getArray`` but if an attempt is made to get a scalar `KeyError` will be raised. (`DM-35396 <https://rubinobs.atlassian.net/browse/DM-35396>`_)
+- ``QuantumGraph`` generation will no longer fail when the ``--dataset-query-constraint`` argument includes a dataset type that is not relevant for one or more pipeline subgraphs. (`DM-47505 <https://rubinobs.atlassian.net/browse/DM-47505>`_)
+
+
 lsst-pipe-base v28.0.0 (2024-11-21)
 ===================================
 

{lsst_pipe_base-29.2025.1200 → lsst_pipe_base-29.2025.1400}/python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

@@ -130,6 +130,10 @@ class AllDimensionsQuantumGraphBuilder(QuantumGraphBuilder):
         tree = _DimensionGroupTree(subgraph)
         self._query_for_data_ids(tree)
         skeleton = self._make_subgraph_skeleton(tree)
+        if not skeleton.has_any_quanta:
+            # QG is going to be empty; exit early not just for efficiency, but
+            # also so downstream code doesn't have to guard against this case.
+            return skeleton
         self._find_followup_datasets(tree, skeleton)
         dimension_records = self._fetch_most_dimension_records(tree)
         leftovers = self._attach_most_dimension_records(skeleton, dimension_records)

{lsst_pipe_base-29.2025.1200 → lsst_pipe_base-29.2025.1400}/python/lsst/pipe/base/quantum_graph_skeleton.py

@@ -194,6 +194,13 @@ class QuantumGraphSkeleton:
         """The total number of edges."""
         return len(self._xgraph.edges)
 
+    @property
+    def has_any_quanta(self) -> bool:
+        """Test whether this graph has any quanta."""
+        for _ in self.iter_all_quanta():
+            return True
+        return False
+
     def has_task(self, task_label: str) -> bool:
         """Test whether the given task is in this skeleton.
 

lsst_pipe_base-29.2025.1400/python/lsst/pipe/base/version.py

@@ -0,0 +1,2 @@
+__all__ = ["__version__"]
+__version__ = "29.2025.1400"

{lsst_pipe_base-29.2025.1200 → lsst_pipe_base-29.2025.1400/python/lsst_pipe_base.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-pipe-base
-Version: 29.2025.1200
+Version: 29.2025.1400
 Summary: Pipeline infrastructure for the Rubin Science Pipelines.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License: BSD 3-Clause License

{lsst_pipe_base-29.2025.1200 → lsst_pipe_base-29.2025.1400}/tests/test_graphBuilder.py

@@ -32,10 +32,21 @@ import logging
 import unittest
 
 import lsst.utils.tests
+from lsst.daf.butler import Butler, DatasetType
 from lsst.daf.butler.registry import UserExpressionError
-from lsst.pipe.base import QuantumGraph
-from lsst.pipe.base.all_dimensions_quantum_graph_builder import DatasetQueryConstraintVariant
+from lsst.pipe.base import PipelineGraph, QuantumGraph
+from lsst.pipe.base.all_dimensions_quantum_graph_builder import (
+    AllDimensionsQuantumGraphBuilder,
+    DatasetQueryConstraintVariant,
+)
 from lsst.pipe.base.tests import simpleQGraph
+from lsst.pipe.base.tests.mocks import (
+    DynamicConnectionConfig,
+    DynamicTestPipelineTask,
+    DynamicTestPipelineTaskConfig,
+    MockDataset,
+    MockStorageClass,
+)
 from lsst.utils.tests import temporaryDirectory
 
 _LOG = logging.getLogger(__name__)
@@ -82,6 +93,63 @@ class GraphBuilderTestCase(unittest.TestCase):
             )
             self.assertEqual(len(qgraph3), 5)
 
+    def test_empty_qg(self):
+        """Test that making an empty QG doesn't raise exceptions."""
+        config = DynamicTestPipelineTaskConfig()
+        config.inputs["i"] = DynamicConnectionConfig(
+            dataset_type_name="input",
+            storage_class="StructuredDataDict",
+            dimensions=["detector"],
+        )
+        config.init_inputs["ii"] = DynamicConnectionConfig(
+            dataset_type_name="init_input",
+            storage_class="StructuredDataDict",
+        )
+        config.outputs["o"] = DynamicConnectionConfig(
+            dataset_type_name="output",
+            storage_class="StructuredDataDict",
+            dimensions=["detector"],
+        )
+        config.init_outputs["io"] = DynamicConnectionConfig(
+            dataset_type_name="init_output",
+            storage_class="StructuredDataDict",
+        )
+        pipeline_graph = PipelineGraph()
+        pipeline_graph.add_task("a", DynamicTestPipelineTask, config)
+        with temporaryDirectory() as repo_path:
+            Butler.makeRepo(repo_path)
+            butler = Butler.from_config(repo_path, writeable=True, run="test_empty_qg")
+            MockStorageClass.get_or_register_mock("StructuredDataDict")
+            butler.registry.registerDatasetType(
+                DatasetType(
+                    "input",
+                    dimensions=butler.dimensions.conform(["detector"]),
+                    storageClass="_mock_StructuredDataDict",
+                )
+            )
+            init_input_dataset_type = DatasetType(
+                "init_input",
+                dimensions=butler.dimensions.empty,
+                storageClass="_mock_StructuredDataDict",
+            )
+            butler.registry.registerDatasetType(init_input_dataset_type)
+            # Init-input initially exists, but input does not (hence empty QG).
+            butler.put(
+                MockDataset(
+                    dataset_id=None,
+                    dataset_type=init_input_dataset_type.to_simple(),
+                    data_id={},
+                    run=butler.run,
+                ),
+                "init_input",
+            )
+            # Attempt to make QG; should just be empty, with no exceptions.
+            self.assertFalse(AllDimensionsQuantumGraphBuilder(pipeline_graph, butler).build())
+            # Initialize the output run, try again, with same expected result.
+            pipeline_graph.register_dataset_types(butler)
+            pipeline_graph.init_output_run(butler)
+            self.assertFalse(AllDimensionsQuantumGraphBuilder(pipeline_graph, butler).build())
+
     # Inconsistent governor dimensions are no longer an error, so this test
     # fails with the new query system. We should probably check instead that
     # logging includes an explanation for the empty QG, but it might not

lsst_pipe_base-29.2025.1200/python/lsst/pipe/base/version.py

@@ -1,2 +0,0 @@
-__all__ = ["__version__"]
-__version__ = "29.2025.1200"