lsst-pipe-base 30.0.0rc2__tar.gz → 30.0.1__tar.gz

{lsst_pipe_base-30.0.0rc2/python/lsst_pipe_base.egg-info → lsst_pipe_base-30.0.1}/PKG-INFO

@@ -1,20 +1,20 @@
 Metadata-Version: 2.4
 Name: lsst-pipe-base
-Version: 30.0.0rc2
+Version: 30.0.1
 Summary: Pipeline infrastructure for the Rubin Science Pipelines.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License-Expression: BSD-3-Clause OR GPL-3.0-or-later
 Project-URL: Homepage, https://github.com/lsst/pipe_base
+Project-URL: Source, https://github.com/lsst/pipe_base
 Keywords: lsst
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering :: Astronomy
-Requires-Python: >=3.11.0
+Requires-Python: >=3.12.0
 Description-Content-Type: text/markdown
 License-File: COPYRIGHT
 License-File: LICENSE

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/CHANGES.rst

@@ -1,3 +1,108 @@
+lsst-pipe-base v30.0.1 (2026-02-03)
+===================================
+
+Dropped support for Python 3.11.
+Tested on Python 3.14.
+
+New Features
+------------
+
+- Added the ``butler ingest-provenance`` command to ingest the provenance quantum graphs written by ``butler aggregate-graph``. (`DM-52738 <https://rubinobs.atlassian.net/browse/DM-52738>`_)
+- Add report tooling (similar to ``pipetask report``, but faster) to the ``ProvenanceQuantumGraph`` class and the ``butler`` command line (as the new ``provenance-report`` subcommand).
+
+  Updatec ``butler ingest-graph`` to delete now-empty config/metadata/log directories.
+
+  Fix a bug that causes provenance quantum graphs to fail to be written if the address file grew to larger than 2G. (`DM-53851 <https://rubinobs.atlassian.net/browse/DM-53851>`_)
+
+
+API Changes
+-----------
+
+- Modified ``Instrument.importAll`` so that it now returns information on the instrument classes that were loaded successfully. (`DM-53882 <https://rubinobs.atlassian.net/browse/DM-53882>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed ``aggregator-graph`` handling of killed subprocesses to avoid hangs.
+
+  Fixed a race condition in ``aggregate-graph`` that could cause hanging at shutdown on small graphs. (`DM-53913 <https://rubinobs.atlassian.net/browse/DM-53913>`_)
+- Fixed a bug that caused provenance recording to fail in ``SeparablePipelineExecutor`` when ``NoWorkFound`` chaining occurs. (`DM-53977 <https://rubinobs.atlassian.net/browse/DM-53977>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- Now guard against quantum graph builds in which the initial query produces many more result rows than end up in the final graph, which can lead to catastrophically large dimension data storage in the QG file. (`DM-53773 <https://rubinobs.atlassian.net/browse/DM-53773>`_)
+- Bumped minimum Python version to 3.12. (`DM-53998 <https://rubinobs.atlassian.net/browse/DM-53998>`_)
+
+
+lsst-pipe-base v30.0.0 (2026-01-16)
+===================================
+
+New Features
+------------
+
+- Added support for healpix (and other non-database dimensions) in quantum graph builder. (`DM-51176 <https://rubinobs.atlassian.net/browse/DM-51176>`_)
+- Added filtering out dataset refs that the destination butler already knows in ``transfer_from_graph`` as well as dividing the transfer into smaller chunks to speed up restarts. (`DM-51273 <https://rubinobs.atlassian.net/browse/DM-51273>`_)
+- Added handling of ``PrerequisiteInput`` in ``QuantaAdjuster``, with a corresponding unit test. (`DM-51509 <https://rubinobs.atlassian.net/browse/DM-51509>`_)
+- Added ``PredictedQuantumGraph``, a replacement for the ``QuantumGraph`` class with more efficient I/O (via a new file format and more partial-read flexibility).
+
+  The new ``PredictedQuantumGraph`` is now the default in most tooling, and the new format can be opted into via the ``.qg`` (instead of ``.qgraph``) file extension.
+  New files can be read with the old class and vice versa.
+
+  The ``QuantumGraph`` class will eventually be deprecated along with much of the current provenance reporting tooling, but only when the new provenance ecosystem is fully in place. (`DM-51850 <https://rubinobs.atlassian.net/browse/DM-51850>`_)
+- Added a ``rename`` dict in ``ImportIR`` to support renaming task labels, with corresponding unit tests. (`DM-52168 <https://rubinobs.atlassian.net/browse/DM-52168>`_)
+- Added the new ``ProvenanceQuantumGraph`` class and the ``aggregate-graph`` tool (a replacement for ``transfer-from-graph``) that writes it at the end of batch runs. (`DM-52360 <https://rubinobs.atlassian.net/browse/DM-52360>`_)
+- Improved provenance tracking for failed quanta and retries.
+
+  By storing extra information in the log datasets written during execution,
+  we can record caught exceptions, track which other quanta have
+  already executed in the same process, and keep track of previous attempts to
+  run the same quantum. (`DM-53019 <https://rubinobs.atlassian.net/browse/DM-53019>`_)
+- Added provenance writing support to ``MPGraphExecutor`` and ``SeparablePipelineExecutor``. (`DM-53622 <https://rubinobs.atlassian.net/browse/DM-53622>`_)
+
+
+API Changes
+-----------
+
+- Moved pipeline executors and their support code here, from ``lsst.ctrl.mpexec``.
+
+  This included minor API changes for ``SingleQuantumExecutor`` as well: consistent snake-case naming, keyword-only arguments for construction, and a switch to private instance attributes. (`DM-48980 <https://rubinobs.atlassian.net/browse/DM-48980>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed ``transfer-from-graph`` to update chain when asked and output run collection exists even if didn't transfer any datasets. (`DM-51821 <https://rubinobs.atlassian.net/browse/DM-51821>`_)
+- Fixed bug in ``transfer_from_graph`` where the input collections were not flattened before adding to a new output chain. (`DM-52004 <https://rubinobs.atlassian.net/browse/DM-52004>`_)
+- Fixed bug where the log's ``MDC.RUN`` was the empty string when using a quantum-backed butler. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
+- Fixed a bug that caused ``PipelineGraph`` objects to be marked as unresolved when loaded from disk. (`DM-52787 <https://rubinobs.atlassian.net/browse/DM-52787>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- The ``Instrument.configPaths`` property can now refer to ``lsst.resources.ResourcePath`` URIs as well as strings (paths or URI strings). (`DM-33226 <https://rubinobs.atlassian.net/browse/DM-33226>`_)
+- Updated pipe_base code to use the constants defined in ``automatic_connection_constants.py``. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
+- Uses UUIDs instead of internal integer IDs in new quantum graph storage.
+
+  This includes backwards compatibility read support for predicted quantum graphs, but not provenance quantum graphs, because those are still experimental anyway.
+
+  This increases the size of the files by ~6%, but it simplifies the codebase and will make consolidating multiple small provenance quantum graphs (as we
+  currently anticipate doing for prompt processing) much more efficient. (`DM-53174 <https://rubinobs.atlassian.net/browse/DM-53174>`_)
+- Used context managers to ensure that database resources are freed. (`DM-53370 <https://rubinobs.atlassian.net/browse/DM-53370>`_)
+- Added more logging for the later steps of QG building. (`DM-53636 <https://rubinobs.atlassian.net/browse/DM-53636>`_)
+
+
+An API Removal or Deprecation
+-----------------------------
+
+- Removes the ``buildExecutionButler`` function and all supporting code.
+
+  Execution butlers (read-only SQLite databases used for batch execution) have been fully superseded by ``lsst.daf.butler.QuantumBackedButler``. (`DM-52044 <https://rubinobs.atlassian.net/browse/DM-52044>`_)
+
+
 lsst-pipe-base v29.1.0 (2025-06-13)
 ===================================
 
@@ -65,7 +170,7 @@ New Features
   Exceptions that lead to task failures are not yet tracked, because we do not write task metadata for failures and hence have nowhere to put the information. (`DM-48536 <https://rubinobs.atlassian.net/browse/DM-48536>`_)
 - Swapped to the new butler query system in ``QuantumGraph`` generation.
 
-  This change should be mostly transparent to users, aside from small changes in speed (typically faster, but not always). (`DM-45896 <https://rubinobs.atlassian.net/browse/DM-45896>`)
+  This change should be mostly transparent to users, aside from small changes in speed (typically faster, but not always). (`DM-45896 <https://rubinobs.atlassian.net/browse/DM-45896>`_)
 
 Bug Fixes
 ---------
@@ -109,7 +214,7 @@ New Features
   For each graph/attempt, the status of each quantum and dataset is recorded in ``QuantumProvenanceGraph.add_new_graph`` and outcomes of quanta over multiple runs are resolved in ``QuantumProvenanceGraph.resolve_duplicates``.
   At the end of this process, we can combine all attempts into a summary.
   This serves to answer the question "What happened to this data ID?" in a holistic sense. (`DM-41711 <https://rubinobs.atlassian.net/browse/DM-41711>`_)
-- Included the number of expected instances in ``pipetask report`` task-level summary for the `QuantumGraphExecutionReport`. (`DM-44368 <https://rubinobs.atlassian.net/browse/DM-44368>`_)
+- Included the number of expected instances in ``pipetask report`` task-level summary for the ``QuantumGraphExecutionReport``. (`DM-44368 <https://rubinobs.atlassian.net/browse/DM-44368>`_)
 - Added mocking support for tasks that write regular datasets with config, log, or metadata storage classes. (`DM-44583 <https://rubinobs.atlassian.net/browse/DM-44583>`_)
 - Added new ``show_dot`` functionality.
 
@@ -185,7 +290,7 @@ New Features
   This interface is available through YAML pipeline specification by specifying the ``labeledSubsetModifyMode`` key when writing YAML import defectives.
 
   New Python interfaces were added for manipulating labeled subsets in a pipeline.
-  These include; ``Pipeline.subsets`` which is a property returning a `dict`` of subset labels to sets of task labels, ``Pipeline.addLabeledSubset`` to add a new labeled subset to a ``Pipeline``, and ``Pipeline.removeLabeledSubset`` to remove a labeled subset from a pipeline. (`DM-41203 <https://rubinobs.atlassian.net/browse/DM-41203>`_)
+  These include; ``Pipeline.subsets`` which is a property returning a `dict` of subset labels to sets of task labels, ``Pipeline.addLabeledSubset`` to add a new labeled subset to a ``Pipeline``, and ``Pipeline.removeLabeledSubset`` to remove a labeled subset from a pipeline. (`DM-41203 <https://rubinobs.atlassian.net/browse/DM-41203>`_)
 - Added ``QuantumGraph`` summary. (`DM-41542 <https://rubinobs.atlassian.net/browse/DM-41542>`_)
 - Added human-readable option to report summary dictionaries. (`DM-41606 <https://rubinobs.atlassian.net/browse/DM-41606>`_)
 - Added a section to pipelines which allows the explicit declaration of which susbsets correspond to steps and the dimensions the step's quanta can be sharded with. (`DM-41650 <https://rubinobs.atlassian.net/browse/DM-41650>`_)

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/creating-a-pipeline.rst

@@ -150,12 +150,12 @@ associated with ``class`` keyword instead of the label directly. The
 the configuration appropriate for this `Pipeline` specified as an additional
 yaml mapping.
 
-The complete complexity of :ref:`lsst.pex.config` can't be represented with simple
+The complete complexity of `lsst.pex.config` can't be represented with simple
 yaml mapping syntax. To account for this, ``config`` blocks in `Pipeline`\ s
 support two special fields: ``file`` and ``python``.
 
 The ``file`` key may be associated with either a single value pointing to a
-filesystem path where a :ref:`lsst.pex.config` file can be found, or a yaml list
+filesystem path where a `lsst.pex.config` file can be found, or a yaml list
 of such paths. The file paths can contain environment variables that will be
 expanded prior to loading the file(s). These files will then be applied to
 the task during configuration time to override any default values.
@@ -477,7 +477,7 @@ desired camera, or can serve as a base for further `Pipeline`\ s to import.
 Command line options for running Pipelines
 ------------------------------------------
 This section is not intended to serve as a tutorial for processing data from
-the command line, for that refer to :ref:`lsst.ctrl.mpexec` or :ref:`lsst.ctrl.bps`.
+the command line, for that refer to `lsst.ctrl.mpexec` or `lsst.ctrl.bps`.
 However, both of these tools accept URI pointers to a `Pipeline`.  These URIs
 can be altered with a specific syntax which will control how the `Pipeline`
 is loaded.

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/creating-a-pipelinetask.rst

@@ -142,7 +142,7 @@ not tied to the exact band passes of an individual telescope filter).
 
 Next, take a look at the fields defined on your new connection class. These
 are defined in a similar way as defining a configuration class, but instead
-of using `~lsst.pex.config.Field` types from :ref:`lsst.pex.config`,
+of using `~lsst.pex.config.Field` types from `lsst.pex.config`,
 connection classes make use of connection types defined in
 :py:mod:`lsst.pipe.base.connectionTypes`. These connections define the inputs and outputs that
 a |PipelineTask| will expect to make use of. Each of these connections documents
@@ -471,9 +471,9 @@ connection had never existed.
 Run-time optional inputs
 ------------------------
 
-A separate mechanism exists that allows an `Input` connection to be made
+A separate mechanism exists that allows an ``Input`` connection to be made
 run-time optional.
-If the `Input.minimum` attribute is initialized to zero for a connection,
+If the ``Input.minimum`` attribute is initialized to zero for a connection,
 graph-building will still generate a quantum, and the `PipelineTask` will
 be run, even if no dataset for that input can be found.
 

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/creating-a-task.rst

@@ -145,7 +145,7 @@ Use the ``__init__`` method (task constructor) to do the following:
 
 - Call the parent task's ``__init__`` method
 - Make subtasks by calling ``self.makeSubtask(name)``, where ``name`` is the name of a field of type `lsst.pex.config.ConfigurableField` in your :ref:`task's configuration <creating-a-task-configuration>`.
-- Make a schema if your task uses an :ref:`lsst.afw.table`.
+- Make a schema if your task uses an `lsst.afw.table`.
   For an example of such a task `lsst.pipe.tasks.calibrate.CalibrateTask`.
 - Initialize any other instance variables your task needs.
 

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/index.rst

@@ -61,6 +61,14 @@ Developing Pipelines
    testing-pipelines-with-mocks.rst
    working-with-pipeline-graphs.rst
 
+Running Pipelines
+-----------------
+
+.. toctree::
+   :maxdepth: 1
+
+   recording-provenance.rst
+
 .. _lsst.pipe.base-contributing:
 
 Contributing
@@ -102,6 +110,10 @@ Python API reference
 
 .. automodapi:: lsst.pipe.base.quantum_graph
 
+.. automodapi:: lsst.pipe.base.quantum_graph.aggregator
+
+.. automodapi:: lsst.pipe.base.quantum_graph.ingest_graph
+
 .. automodapi:: lsst.pipe.base.quantum_graph.visualization
 
 QuantumGraph generation API reference

lsst_pipe_base-30.0.1/doc/lsst.pipe.base/recording-provenance.rst

@@ -0,0 +1,108 @@
+.. _pipe_base_provenance:
+
+.. py:currentmodule:: lsst.pipe.base.quantum_graph
+
+####################
+Recording Provenance
+####################
+
+The `PredictedQuantumGraph` that is used to predict and control processing also contains a wealth of provenance information, including task configuration and the complete input-output relationships between all datasets.
+Instead of storing these graphs directly in a `~lsst.daf.butler.Butler` repository, however, it is better to first augment them with additional provenance information that is only available after execution has completed, producing a `ProvenanceQuantumGraph` that is ingested instead.
+We store provenance in a ``run_provenance`` dataset type with empty dimensions, which means there is exactly one for each `~lsst.daf.butler.CollectionType.RUN` collection.
+In addition to the input-output graph itself and full configuration for all tasks, `ProvenanceQuantumGraph` stores status information for each attempt to run a quantum, including exception information and caveats on any successes.
+It can also store the full logs and task metadata for each quantum, allowing repositories to store many fewer small files (it is possible to continue to have per-quantum butler datasets for these, all backed by the same file).
+
+The pipeline system has many different execution contexts, and provenance recording is not supported in all of them at this time.
+
+Batch Execution / Quantum-Backed Butler
+=======================================
+
+Provenance recording is fully supported in batch workflows that use the `~lsst.daf.butler.QuantumBackedButler` class (e.g. ``pipetask run-qbb``, as run by the ``bps`` tool) to avoid database writes during execution.
+This involves the following steps:
+
+- A `PredictedQuantumGraph` is generated as usual (e.g. via ``pipetask qgraph``, as run by ``bps submit``) and saved to a known location.
+- All quanta are executed via ``pipetask run-qbb``, writing their outputs to butler-managed storage without updating the butler database.
+- When all quanta have been attempted, the ``butler aggregate-graph`` tool is run (e.g. in the BPS ``finalJob``) to ingest output datasets into the butler database, and the ``--output`` option is used to save a `ProvenanceQuantumGraph` to a known location.
+  This step and the previous one may be run multiple times (e.g. via ``bps restart``) to retry some failures, and it is only necessary to pass ``--output`` the last time (though usually the user does not know which attempt will be the last one).
+- When all processing attempts are complete, the ``butler ingest-graph`` tool is used to ingest the graph into the butler database and rewrite all metadata, log, and config datasets to also be backed by the same graph file (deleting the original files).
+  This step should not be included in the BPS ``finalJob`` (see below).
+
+All of the above happens in a single `~lsst.daf.butler.CollectionType.RUN` collection.
+Reference documentation for ``butler aggregate-graph`` and ``butler ingest-graph`` can be found in the `aggregator` and `ingest_graph` modules that implement them (respectively); in both cases there are Python interfaces that closely mirror the command-line ones.
+
+Parallelization
+---------------
+
+Aggregating and ingesting a large batch run is expensive, and both tools use parallelism whenever possible to improve performance.
+
+The aggregator in particular is explicitly parallel, with separate workers (usually subprocesses) assigned to scan and read metadata and log files (any number of workers), ingest datasets (a single worker), write the provenance graph file (a single worker), and coordinate all of these operations.
+Since all information must be passed from the scanners to the ingestion and writer workers, additional parallelism can help when all operations are running at around the same speed (as reported in the logs), but not when ingestion or writing lags significantly behind.
+The writer process has substantial startup overhead and will typically lag the others at the beginning before it catches up later.
+
+The `ingest_graph` tool mostly performs database write operations, which do not benefit from parallelism, but it also deletes the original metadata, log, and config files as the new graph-backed variants of those datasets are ingested.
+These deletes are delegated to `lsst.resources.ResourcePath.mremove`, which refers to the ``LSST_RESOURCES_NUM_WORKERS``, ``LSST_RESOURCES_EXECUTOR``, and ``LSST_S3_USE_THREADS`` environment variables to control parallelism.
+As with other butler bulk-delete operations, the default parallelism is usually fine.
+
+.. note::
+
+  Earlier versions of the `aggregator` would run catastrophically slowly when ``LSST_RESOURCES_EXECUTOR=process``, as this made each scanner process spawn multiple subprocesses constantly.
+  In recent versions all parallelism environment variables are ignored by the aggregator so this should not occur.
+
+Ingesting Outputs Early
+-----------------------
+
+The `aggregator` may be run with `~aggregator.AggregatorConfig.incomplete` set to `True` (``--incomplete`` on the command line) to allow it to be safely run before the graph has finished executing.
+Note that while ingestion always picks up where it left off, scanning always has to start at the beginning, and provenance graph writing is disabled when running in ``incomplete`` mode, so while this allows output datasets be be available via the `~lsst.daf.butler.Butler` sooner, it does not generally make the final complete `aggregator` call substantially faster.
+
+Promising Graph Ingestion
+-------------------------
+
+By default, the `aggregator` ingests all metadata, log, and config outputs into the butler database in the usual way, i.e. backed by their original individual files.
+The `ingest_graph` tool then has to delete these datasets from the butler database before it can ingest new ones and delete the original files.
+When it is known in advance that `ingest_graph` will be run later, the `~aggregator.AggregatorConfig.promise_ingest_graph` (``--promise-ingest-graph``) option can be used to tell the `aggregator` *not* to ingest these, saving time for both commands.
+This option must be used with care, however: if `ingest_graph` isn't run later, the original files will be orphaned in a butler-managed location without any record in the database, which generally means they'll quietly take up space.
+In addition, because the metadata datasets are used by the middleware system as the indicator of a quantum's success, their absence will make any downstream quantum graphs built using ``--skip-existing-in`` incorrect.
+And of course any downstream quantum graph builds that actually use those datasets as input (only metadata should be) will not see them as available.
+
+Deferring Graph Ingestion
+-------------------------
+
+Ingesting the provenance graph is not generally necessary to kick off downstream processing by building new quantum graphs for later pipeline steps, and it is always safe to build downstream quantum graphs if `~aggregator.AggregatorConfig.promise_ingest_graph` is left `False`.
+It can also be done safely if `~aggregator.AggregatorConfig.promise_ingest_graph` is `True` and:
+
+ - ``--skip-existing-in`` is not used;
+ - the downstream processing does not use metadata, log, or config datasets as an overall input (``pipetask build ... --show inputs`` can be used to check for this).
+
+These conditions also must be met in order for `ingest_graph` to be safely run *while* a downstream quantum graph is being executed.
+Both of these conditions are *usually* met, and deferring and promising graph ingest each provide significant wall-clock savings, so we recommend the following approach for very large BPS campaigns:
+
+- Submit ``step(N)`` to BPS with ``--promise-ingest-graph`` in the ``finalJob`` invocation of ``aggregate-graph``.
+- When ready to move on to ``step(N+1)``, run ``pipetask build ... --show inputs`` (on ``step(N+1)``) to scan for metadata, log, and config inputs that may be needed from the previous step.
+- If there are no such inputs, immediately submit that step to BPS, and run `ingest_graph` on ``step(N)`` as soon as the quantum graph for ``step(N+1)`` is built (it could be built at the same time, but waiting a bit may help spread out database load).
+- If there are metadata, log, or config inputs, run `ingest_graph` on ``step(N)`` and wait for it to finish before submitting ``step(N+1)``.
+
+Note that *independent* quantum graph builds (e.g. same tasks, disjoint data IDs) can always be built before or while `ingest_graph` runs.
+
+Recovering from Interruptions
+-----------------------------
+
+If the `aggregator` is interrupted it can simply be started again.
+Database ingestion will pick up where it left off, while scanning and provenance-graph writing will start over from the beginning.
+
+If `ingest_graph` is interrupted, it can also be started again, and everything will pick up where it left off.
+To guarantee this it always modifies the repository in the following order:
+
+- if the ``run_provenance`` dataset does not exist in the collection, all existing metadata/log/config datasets are assumed to be backed by their original files and are deleted from the butler database (without deleting the files);
+- the ``run_provenance`` dataset itself is ingested (this ensures the metadata/log/config *content* is safe inside the butler, even if it's not fully accessible);
+- in batches, metadata/log/config datasets are reingested into the butler backed by the graph file, and then the corresponding original files are deleted.
+
+This means we can use the existence of ``run_provenance`` and any particular metadata/log/config dataset in the butler database to infer the status of the original files.
+
+In fact, if `ingest_graph` is interrupted at any point, it *must* be tried again until it succeeds, since not doing so can leave metadata/log/config files orphaned, just like when `~aggregator.AggregatorConfig.promise_ingest_graph` is `True`.
+
+.. note::
+
+  After the ``run_provenance`` dataset is ingested, it is *not* safe to run the `aggregator`: the `aggregator` reads the original metadata and log files to gather provenance information, and will infer the wrong states for quanta if those are missing because `ingest_graph` has deleted them.
+
+  This is why it is not safe to run ``bps restart`` after `ingest_graph`, and why we do not recommend adding `ingest_graph` to the BPS ``finalJob``, even if the user is willing to forgo using ``bps restart``: by default, the ``finalJob`` will be retried on failure, causing the `aggregator` to run again when it may not be safe to do so.
+  And if ``finalJob`` retries are disabled, it is too easy for the repository to end up in a state that would require manual `ingest_graph` runs to prevent orphan datasets.

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/task-framework-overview.rst

@@ -25,14 +25,14 @@ A common use for this is to provide a camera-specific variant of a particular ta
 Tasks may process multiple items of data in parallel, using the ``ctrl_mpexec`` package and its ``pipetask`` command.
 
 Most tasks have a ``run`` method that performs the primary data processing.
-Each task's `run` method should return a `~lsst.pipe.base.Struct`.
+Each task's `~lsst.pipe.base.Task.run` method should return a `~lsst.pipe.base.Struct`.
 This allows named access to returned data, which provides safer evolution than relying on the order of returned values.
 All task methods that return more than one or two items of data should return the data in a `~lsst.pipe.base.Struct`.
 
 Many tasks are found in the ``pipe_tasks`` package, especially tasks that use many different packages and don't seem to belong in any one of them.
-Tasks that are associated with a particular package should be in that package; for example the instrument signature removal task ``ip.isr.isrTask.IsrTask`` is in the ``ip_isr`` package.
+Tasks that are associated with a particular package should be in that package; for example the instrument signature removal task ``lsst.ip.isr.isrTask.IsrTask`` is in the ``ip_isr`` package.
 
-`pipe_base` is written purely in Python. The most important contents are:
+``pipe_base`` is written purely in Python. The most important contents are:
 
 - `~lsst.pipe.base.PipelineTask`: base class for pipeline tasks that can be run from pipelines.
 - `~lsst.pipe.base.Task`: base class for subtasks that are not meant to be run from the

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/doc/lsst.pipe.base/working-with-pipeline-graphs.rst

@@ -64,7 +64,7 @@ In some cases it may be more convenient to add tasks to an existing `PipelineGra
 
 **The most important thing to remember when modifying `PipelineGraph` objects is that modifications typically reset some or all of the graph to an unresolved state.**
 
-The reference documentation for these methods describes exactly what guarantees they make about existing resolutions in detail, and what operations are still supported on unresolved or partially-resolved graphs, but it is easiest to just ensure `resolve` is called after any modifications are complete.
+The reference documentation for these methods describes exactly what guarantees they make about existing resolutions in detail, and what operations are still supported on unresolved or partially-resolved graphs, but it is easiest to just ensure ``resolve`` is called after any modifications are complete.
 
 `PipelineGraph` mutator methods provide strong exception safety (the graph is left unchanged when an exception is raised and caught by calling code) unless the exception type raised is `PipelineGraphExceptionSafetyError`.
 

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "lsst-pipe-base"
-requires-python = ">=3.11.0"
+requires-python = ">=3.12.0"
 description = "Pipeline infrastructure for the Rubin Science Pipelines."
 license = "BSD-3-Clause OR GPL-3.0-or-later"
 license-files = ["COPYRIGHT", "LICENSE", "bsd_license.txt", "gpl-v3.0.txt"]
@@ -16,7 +16,6 @@ classifiers = [
     "Intended Audience :: Science/Research",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: 3.14",
@@ -45,6 +44,7 @@ pipe_base = "lsst.pipe.base.cli:get_cli_subcommands"
 
 [project.urls]
 "Homepage" = "https://github.com/lsst/pipe_base"
+"Source" = "https://github.com/lsst/pipe_base"
 
 [project.optional-dependencies]
 test = ["pytest >= 3.2"]
@@ -101,15 +101,6 @@ version = { attr = "lsst_versions.get_lsst_version" }
         name = "An API Removal or Deprecation"
         showcontent = true
 
-[tool.black]
-line-length = 110
-target-version = ["py311"]
-
-[tool.isort]
-profile = "black"
-line_length = 110
-known_first_party = ["lsst"]
-
 [tool.lsst_versions]
 write_to = "python/lsst/pipe/base/version.py"
 
@@ -140,7 +131,7 @@ add-ignore = ["D107", "D105", "D102", "D100", "D200", "D205", "D400", "D104"]
 
 [tool.ruff]
 line-length = 110
-target-version = "py311"
+target-version = "py312"
 exclude = [
     "__init__.py",
 ]
@@ -204,6 +195,7 @@ checks = [
     "RT01",  # Unfortunately our @property trigger this.
     "RT02",  # Does not want named return value. DM style says we do.
     "SS05",  # pydocstyle is better at finding infinitive verb.
+    "RT03",  # sphinx bug requiring empty Returns.
 ]
 exclude = [
     '^__init__$',

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/python/lsst/pipe/base/_instrument.py

@@ -31,14 +31,14 @@ __all__ = ("Instrument",)
 
 import contextlib
 import datetime
-import os.path
 from abc import ABCMeta, abstractmethod
 from collections.abc import Sequence
 from typing import TYPE_CHECKING, Any, Self, cast, final
 
-from lsst.daf.butler import DataCoordinate, DataId, DimensionPacker, DimensionRecord, Formatter
+from lsst.daf.butler import DataCoordinate, DataId, DimensionPacker, DimensionRecord, Formatter, FormatterV2
 from lsst.daf.butler.registry import DataIdError
 from lsst.pex.config import Config, RegistryField
+from lsst.resources import ResourcePath, ResourcePathExpression
 from lsst.utils import doImportType
 from lsst.utils.introspection import get_full_type_name
 
@@ -65,10 +65,10 @@ class Instrument(metaclass=ABCMeta):
     the base class.
     """
 
-    configPaths: Sequence[str] = ()
+    configPaths: Sequence[ResourcePathExpression] = ()
     """Paths to config files to read for specific Tasks.
 
-    The paths in this list should contain files of the form `task.py`, for
+    The paths in this list should contain files of the form ``task.py``, for
     each of the Tasks that requires special configuration.
     """
 
@@ -99,7 +99,8 @@ class Instrument(metaclass=ABCMeta):
 
     @abstractmethod
     def register(self, registry: Registry, *, update: bool = False) -> None:
-        """Insert instrument, and other relevant records into `Registry`.
+        """Insert instrument, and other relevant records into a butler
+        registry.
 
         Parameters
         ----------
@@ -109,6 +110,10 @@ class Instrument(metaclass=ABCMeta):
             If `True` (`False` is default), update existing records if they
             differ from the new ones.
 
+        Returns
+        -------
+        None
+
         Raises
         ------
         lsst.daf.butler.registry.ConflictingDefinitionError
@@ -127,13 +132,6 @@ class Instrument(metaclass=ABCMeta):
         the level of individual dimension entries; new detectors and filters
         should be added, but changes to any existing record should not be.
         This can generally be achieved via a block like
-
-        .. code-block:: python
-
-            with registry.transaction():
-                registry.syncDimensionData("instrument", ...)
-                registry.syncDimensionData("detector", ...)
-                self.registerFilters(registry)
         """
         raise NotImplementedError()
 
@@ -314,7 +312,7 @@ class Instrument(metaclass=ABCMeta):
         return instrument_cls(collection_prefix=collection_prefix)
 
     @staticmethod
-    def importAll(registry: Registry) -> None:
+    def importAll(registry: Registry) -> dict[str, type[Instrument]]:
         """Import all the instruments known to this registry.
 
         This will ensure that all metadata translators have been registered.
@@ -324,31 +322,43 @@ class Instrument(metaclass=ABCMeta):
         registry : `lsst.daf.butler.Registry`
             Butler registry to query to find the information.
 
+        Returns
+        -------
+        imported : `dict` [`str`, `type` [`Instrument`]]
+            A mapping containing all the instrument classes that were loaded
+            successfully, keyed by their butler names.
+
         Notes
         -----
         It is allowed for a particular instrument class to fail on import.
         This might simply indicate that a particular obs package has
         not been setup.
         """
+        imported: dict[str, type[Instrument]] = {}
         records = list(registry.queryDimensionRecords("instrument"))
         for record in records:
             cls = record.class_name
+            instrument_name: str = cast(str, record.name)
             with contextlib.suppress(Exception):
-                doImportType(cls)
+                instr = doImportType(cls)
+                assert issubclass(instr, Instrument)
+                imported[instrument_name] = instr
+        return imported
 
     @abstractmethod
-    def getRawFormatter(self, dataId: DataId) -> type[Formatter]:
+    def getRawFormatter(self, dataId: DataId) -> type[Formatter | FormatterV2]:
         """Return the Formatter class that should be used to read a particular
         raw file.
 
         Parameters
         ----------
-        dataId : `DataId`
+        dataId : `lsst.daf.butler.DataId`
             Dimension-based ID for the raw file or files being ingested.
 
         Returns
         -------
-        formatter : `lsst.daf.butler.Formatter` class
+        formatter : `type` \
+                [`lsst.daf.butler.Formatter` | `lsst.daf.butler.FormatterV2` ]
             Class to be used that reads the file into the correct
             Python object for the raw data.
         """
@@ -366,9 +376,10 @@ class Instrument(metaclass=ABCMeta):
             Config instance to which overrides should be applied.
         """
         for root in self.configPaths:
-            path = os.path.join(root, f"{name}.py")
-            if os.path.exists(path):
-                config.load(path)
+            resource = ResourcePath(root, forceDirectory=True, forceAbsolute=True)
+            uri = resource.join(f"{name}.py", forceDirectory=False)
+            if uri.exists():
+                config.load(uri)
 
     @staticmethod
     def formatCollectionTimestamp(timestamp: str | datetime.datetime) -> str:

{lsst_pipe_base-30.0.0rc2 → lsst_pipe_base-30.0.1}/python/lsst/pipe/base/_quantumContext.py

@@ -380,8 +380,8 @@ class QuantumContext:
             if dataset is directly a `list` of `~lsst.daf.butler.DatasetRef`
             or a single `~lsst.daf.butler.DatasetRef`. If ``values.NAME`` is
             None, no output is written.
-        dataset : `OutputQuantizedConnection` or `list`[`DatasetRef`] \
-                or `DatasetRef`
+        dataset : `OutputQuantizedConnection` or `list` \
+                [`lsst.daf.butler.DatasetRef`] or `lsst.daf.butler.DatasetRef`
             This argument may either be an `InputQuantizedConnection` which
             describes all the inputs of a quantum, a list of
             `lsst.daf.butler.DatasetRef`, or a single
@@ -460,7 +460,7 @@ class QuantumContext:
 
         Parameters
         ----------
-        ref : `DatasetRef`
+        ref : `lsst.daf.butler.DatasetRef`
             The dataset to attach provenance to. This dataset must have been
             retrieved by this quantum context.
         extra : `dict` [ `str`, `int` | `float` | `str` | `bool` ]