lsst-pipe-base 29.2025.4100__py3-none-any.whl → 29.2025.4300__py3-none-any.whl

lsst/pipe/base/quantum_graph/aggregator/__init__.py

@@ -0,0 +1,143 @@
+# This file is part of pipe_base.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (http://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This software is dual licensed under the GNU General Public License and also
+# under a 3-clause BSD license. Recipients may choose which of these licenses
+# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
+# respectively.  If you choose the GPL option then the following text applies
+# (but note that there is still no warranty even if you opt for BSD instead):
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""A tool for aggregating provenance information, logs, and metadata from a
+processing run into a `ProvenanceQuantumGraph`.
+
+The aggregator uses multiple processes (typically) or threads to gather
+information about a processing run that has already completed, ingest the
+output datasets into a butler repository, and write provenance information to
+disk.
+
+The only public entry point is the `aggregate_graph` function, with all options
+controlled by an `AggregatorConfig` instance.
+Usually the tool is invoked from the command line as
+``butler aggregate-graph``.
+
+Most log messages are sent to the ``aggregate-graph`` log, which users may want
+to set to ``VERBOSE`` to get additional detail, including periodic logging.
+See `AggregatorConfig` for how to enable additional logging by each process.
+"""
+
+from __future__ import annotations
+
+__all__ = ("AggregatorConfig", "FatalWorkerError", "aggregate_graph")
+
+from ._config import AggregatorConfig
+from ._supervisor import aggregate_graph
+from ._communicators import FatalWorkerError
+
+#
+# Aggregator Design Notes and Package Layout
+# ==========================================
+#
+# The aggregator tool is designed to run in multiple processes.  Using threads
+# instead is supported, but mostly as a way to leverage the GIL to (mostly)
+# only occupy a single core without having to maintain two largely distinct
+# implementations.  Throughout the rest of these notes I'll just assume they're
+# processes.
+#
+# There are four types of processes, each represented in code as a class in its
+# own module:
+#
+# - The Supervisor runs in the main process and is responsible for interacting
+#   with the user (reporting progress, handling KeyboardInterrupt) and
+#   traversing the predicted quantum graph.
+#
+# - Scanner workers are responsible for reading log and metadata files using a
+#   QBB, occasionally checking for the existence of other outputs (we can
+#   usually get the information about which outputs exist from the metadata)
+#   and sending the information we extract from them to other workers. Scanners
+#   are the only kind of worker we can add more of when we are given more
+#   cores, so we try to do as much pre-processing as possible on them (e.g.
+#   compression, datastore record extraction).
+#
+# - The ingester worker is responsible for ingesting output datasets into the
+#   central butler repository.  It batches up ingest requests from the scanners
+#   until the batch size exceeds a configurable limit (e.g. 10k datasets), and
+#   then ingests those pending requests in a single transaction.
+#
+# - The writer worker is responsible for actually writing the
+#   `ProvenanceQuantumGraph` file.  Unlike the other workers, it needs to read
+#   almost the entire predicted quantum graph file before it starts, so it can
+#   take a few minutes before it starts its main loop of accepting write
+#   requests from the scanners (most of the time) and the supervisor (only for
+#   blocked quanta, which we don't bother to scan).  For each of the big
+#   multi-block files (datasets, quanta, logs, metadata) the writes opens and
+#   writes to a temporary file, which is moved into the provenance QG zip
+#   archive at the end.  In addition, to try to keep expensive operations on
+#   the scanners, the writer builds a ZStandard compression dictionary from the
+#   first write requests it sees, and then sends that to the scanners so they
+#   can take over doing the compression after that is complete. Since writing
+#   the provenance graph is (at present) optional, sometimes there is no writer
+#   worker.
+#
+# Each process is also associated with its own "communicator" class, which are
+# all defined together in the _communicators.py file.  The communicators manage
+# the queues that are used to send information between processes, as well as
+# providing loggers and some other common utility functionality. Communicators
+# are context managers that take care of cleanly shutting down each process
+# when it completes or is interrupted; `multiprocessing.Queue` objects need to
+# be cleared out carefully or you get deadlocks that prevent processes from
+# actually exiting, and so a lot of the work of the communicators involves
+# passing sentinal objects over those queues so we can identify when they
+# really are empty.
+#
+# The other modules in the packages include:
+#
+# - _config.py: the `AggregatorConfig` model, which is essentially the public
+#   interface for controlling the aggregator.
+#
+# - _progress.py: helper objects for [periodic] logging and tqdm progress bars.
+#   It's not clear the tqdm stuff should continue to exist once development of
+#   the tool is complete and we stop wanting to run it from an interactive
+#   terminal, but it's certainly been useful for development so far.
+#
+# - _structs.py: simple dataclasses used by multiple workers (most of what we
+#   put on the queues to request scans/ingests/writes);
+#
+# The main logger used by the aggregator is called
+# ``lsst.pipe.base.quantum_graph.aggregator``, and it emits at ``VERBOSE`` and
+# above.  At ``VERBOSE``, it's also used for periodic reporting of status
+# (unless the tqdm progress bars are enabled instead). Each worker also has its
+# own log (including the supervisor) with names like
+# ``lsst.pipe.base.quantum_graph.aggregator.scanner-012`` that include
+# additional information at ``DEBUG`` and above, but these are *not* propagated
+# up; instead there's a config option to write per-worker logs to separate
+# files in a directory.  It would probably be more idiomatic to set up a
+# QueueHandler to feed all the LogRecords the supervisor and use Python logging
+# configuration to send them wherever desired, but::
+#
+# - that seemed to require more wading into `lsst.daf.butler.cli.cliLog` a lot
+#   more than I cared to (especially considering that we want this to work in
+#   `threading` and `multiprocessing`, and logging configuration is very
+#   different in those two contexts);
+#
+# - having the worker logs go to separate files is actually very nice, and it's
+#   more efficient if they just do that themselves, and that's not something
+#   our logging CLI can actually do, AFAICT.
+