lsst-pipe-base 30.0.0__tar.gz → 30.0.0rc2__tar.gz

{lsst_pipe_base-30.0.0/python/lsst_pipe_base.egg-info → lsst_pipe_base-30.0.0rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-pipe-base
-Version: 30.0.0
+Version: 30.0.0rc2
 Summary: Pipeline infrastructure for the Rubin Science Pipelines.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License-Expression: BSD-3-Clause OR GPL-3.0-or-later

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/doc/lsst.pipe.base/CHANGES.rst

@@ -1,69 +1,3 @@
-lsst-pipe-base v30.0.0 (2026-01-16)
-===================================
-
-New Features
-------------
-
-- Added support for healpix (and other non-database dimensions) in quantum graph builder. (`DM-51176 <https://rubinobs.atlassian.net/browse/DM-51176>`_)
-- Added filtering out dataset refs that the destination butler already knows in ``transfer_from_graph`` as well as dividing the transfer into smaller chunks to speed up restarts. (`DM-51273 <https://rubinobs.atlassian.net/browse/DM-51273>`_)
-- Added handling of ``PrerequisiteInput`` in ``QuantaAdjuster``, with a corresponding unit test. (`DM-51509 <https://rubinobs.atlassian.net/browse/DM-51509>`_)
-- Added ``PredictedQuantumGraph``, a replacement for the ``QuantumGraph`` class with more efficient I/O (via a new file format and more partial-read flexibility).
-
-  The new ``PredictedQuantumGraph`` is now the default in most tooling, and the new format can be opted into via the ``.qg`` (instead of ``.qgraph``) file extension.
-  New files can be read with the old class and vice versa.
-
-  The ``QuantumGraph`` class will eventually be deprecated along with much of the current provenance reporting tooling, but only when the new provenance ecosystem is fully in place. (`DM-51850 <https://rubinobs.atlassian.net/browse/DM-51850>`_)
-- Added a ``rename`` dict in ``ImportIR`` to support renaming task labels, with corresponding unit tests. (`DM-52168 <https://rubinobs.atlassian.net/browse/DM-52168>`_)
-- Added the new ``ProvenanceQuantumGraph`` class and the ``aggregate-graph`` tool (a replacement for ``transfer-from-graph``) that writes it at the end of batch runs. (`DM-52360 <https://rubinobs.atlassian.net/browse/DM-52360>`_)
-- Improved provenance tracking for failed quanta and retries.
-
-  By storing extra information in the log datasets written during execution,
-  we can record caught exceptions, track which other quanta have
-  already executed in the same process, and keep track of previous attempts to
-  run the same quantum. (`DM-53019 <https://rubinobs.atlassian.net/browse/DM-53019>`_)
-- Added provenance writing support to ``MPGraphExecutor`` and ``SeparablePipelineExecutor``. (`DM-53622 <https://rubinobs.atlassian.net/browse/DM-53622>`_)
-
-
-API Changes
------------
-
-- Moved pipeline executors and their support code here, from ``lsst.ctrl.mpexec``.
-
-  This included minor API changes for ``SingleQuantumExecutor`` as well: consistent snake-case naming, keyword-only arguments for construction, and a switch to private instance attributes. (`DM-48980 <https://rubinobs.atlassian.net/browse/DM-48980>`_)
-
-
-Bug Fixes
----------
-
-- Fixed ``transfer-from-graph`` to update chain when asked and output run collection exists even if didn't transfer any datasets. (`DM-51821 <https://rubinobs.atlassian.net/browse/DM-51821>`_)
-- Fixed bug in ``transfer_from_graph`` where the input collections were not flattened before adding to a new output chain. (`DM-52004 <https://rubinobs.atlassian.net/browse/DM-52004>`_)
-- Fixed bug where the log's ``MDC.RUN`` was the empty string when using a quantum-backed butler. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
-- Fixed a bug that caused ``PipelineGraph`` objects to be marked as unresolved when loaded from disk. (`DM-52787 <https://rubinobs.atlassian.net/browse/DM-52787>`_)
-
-
-Other Changes and Additions
----------------------------
-
-- The ``Instrument.configPaths`` property can now refer to ``lsst.resources.ResourcePath`` URIs as well as strings (paths or URI strings). (`DM-33226 <https://rubinobs.atlassian.net/browse/DM-33226>`_)
-- Updated pipe_base code to use the constants defined in ``automatic_connection_constants.py``. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
-- Uses UUIDs instead of internal integer IDs in new quantum graph storage.
-
-  This includes backwards compatibility read support for predicted quantum graphs, but not provenance quantum graphs, because those are still experimental anyway.
-
-  This increases the size of the files by ~6%, but it simplifies the codebase and will make consolidating multiple small provenance quantum graphs (as we
-  currently anticipate doing for prompt processing) much more efficient. (`DM-53174 <https://rubinobs.atlassian.net/browse/DM-53174>`_)
-- Used context managers to ensure that database resources are freed. (`DM-53370 <https://rubinobs.atlassian.net/browse/DM-53370>`_)
-- Added more logging for the later steps of QG building. (`DM-53636 <https://rubinobs.atlassian.net/browse/DM-53636>`_)
-
-
-An API Removal or Deprecation
------------------------------
-
-- Removes the ``buildExecutionButler`` function and all supporting code.
-
-  Execution butlers (read-only SQLite databases used for batch execution) have been fully superseded by ``lsst.daf.butler.QuantumBackedButler``. (`DM-52044 <https://rubinobs.atlassian.net/browse/DM-52044>`_)
-
-
 lsst-pipe-base v29.1.0 (2025-06-13)
 ===================================
 

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/_instrument.py

@@ -31,6 +31,7 @@ __all__ = ("Instrument",)
 
 import contextlib
 import datetime
+import os.path
 from abc import ABCMeta, abstractmethod
 from collections.abc import Sequence
 from typing import TYPE_CHECKING, Any, Self, cast, final
@@ -38,7 +39,6 @@ from typing import TYPE_CHECKING, Any, Self, cast, final
 from lsst.daf.butler import DataCoordinate, DataId, DimensionPacker, DimensionRecord, Formatter
 from lsst.daf.butler.registry import DataIdError
 from lsst.pex.config import Config, RegistryField
-from lsst.resources import ResourcePath, ResourcePathExpression
 from lsst.utils import doImportType
 from lsst.utils.introspection import get_full_type_name
 
@@ -65,7 +65,7 @@ class Instrument(metaclass=ABCMeta):
     the base class.
     """
 
-    configPaths: Sequence[ResourcePathExpression] = ()
+    configPaths: Sequence[str] = ()
     """Paths to config files to read for specific Tasks.
 
     The paths in this list should contain files of the form `task.py`, for
@@ -366,10 +366,9 @@ class Instrument(metaclass=ABCMeta):
             Config instance to which overrides should be applied.
         """
         for root in self.configPaths:
-            resource = ResourcePath(root, forceDirectory=True, forceAbsolute=True)
-            uri = resource.join(f"{name}.py", forceDirectory=False)
-            if uri.exists():
-                config.load(uri)
+            path = os.path.join(root, f"{name}.py")
+            if os.path.exists(path):
+                config.load(path)
 
     @staticmethod
     def formatCollectionTimestamp(timestamp: str | datetime.datetime) -> str:

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/log_capture.py

@@ -31,15 +31,17 @@ __all__ = ["LogCapture"]
 
 import dataclasses
 import logging
+import os
+import shutil
+import tempfile
 import uuid
 from collections.abc import Iterator
-from contextlib import contextmanager
+from contextlib import contextmanager, suppress
 from logging import FileHandler
 
 import pydantic
 
-from lsst.daf.butler import Butler, LimitedButler, Quantum
-from lsst.daf.butler._rubin.temporary_for_ingest import TemporaryForIngest
+from lsst.daf.butler import Butler, FileDataset, LimitedButler, Quantum
 from lsst.daf.butler.logging import (
     ButlerLogRecord,
     ButlerLogRecordHandler,
@@ -163,9 +165,7 @@ class LogCapture:
         return cls(butler, butler)
 
     @contextmanager
-    def capture_logging(
-        self, task_node: TaskNode, /, quantum: Quantum, records: ButlerLogRecords | None = None
-    ) -> Iterator[_LogCaptureContext]:
+    def capture_logging(self, task_node: TaskNode, /, quantum: Quantum) -> Iterator[_LogCaptureContext]:
         """Configure logging system to capture logs for execution of this task.
 
         Parameters
@@ -174,9 +174,6 @@ class LogCapture:
             The task definition.
         quantum : `~lsst.daf.butler.Quantum`
             Single Quantum instance.
-        records : `lsst.daf.butler.logging.ButlerLogRecords`, optional
-            Log record container to append to and save.  If provided, streaming
-            mode is disabled (since we'll be saving logs in memory anyway).
 
         Notes
         -----
@@ -208,40 +205,44 @@ class LogCapture:
 
         # Add a handler to the root logger to capture execution log output.
         if log_dataset_name is not None:
-            try:
-                [ref] = quantum.outputs[log_dataset_name]
-            except LookupError as exc:
-                raise InvalidQuantumError(
-                    f"Quantum outputs is missing log output dataset type {log_dataset_name};"
-                    " this could happen due to inconsistent options between QuantumGraph generation"
-                    " and execution"
-                ) from exc
             # Either accumulate into ButlerLogRecords or stream JSON records to
             # file and ingest that (ingest is possible only with full butler).
-            if self.stream_json_logs and self.full_butler is not None and records is None:
-                with TemporaryForIngest(self.full_butler, ref) as temporary:
-                    log_handler_file = FileHandler(temporary.ospath)
-                    log_handler_file.setFormatter(JsonLogFormatter())
-                    logging.getLogger().addHandler(log_handler_file)
-
-                    try:
-                        with ButlerMDC.set_mdc(mdc):
-                            yield ctx
-                    finally:
-                        # Ensure that the logs are stored in butler.
-                        logging.getLogger().removeHandler(log_handler_file)
-                        log_handler_file.close()
-                        if ctx.extra:
-                            with open(temporary.ospath, "a") as log_stream:
-                                ButlerLogRecords.write_streaming_extra(
-                                    log_stream,
-                                    ctx.extra.model_dump_json(exclude_unset=True, exclude_defaults=True),
-                                )
-                        if ctx.store:
-                            temporary.ingest()
+            if self.stream_json_logs and self.full_butler is not None:
+                # Create the log file in a temporary directory rather than
+                # creating a temporary file. This is necessary because
+                # temporary files are created with restrictive permissions
+                # and during file ingest these permissions persist in the
+                # datastore. Using a temp directory allows us to create
+                # a file with umask default permissions.
+                tmpdir = tempfile.mkdtemp(prefix="butler-temp-logs-")
+
+                # Construct a file to receive the log records and "touch" it.
+                log_file = os.path.join(tmpdir, f"butler-log-{task_node.label}.json")
+                with open(log_file, "w"):
+                    pass
+                log_handler_file = FileHandler(log_file)
+                log_handler_file.setFormatter(JsonLogFormatter())
+                logging.getLogger().addHandler(log_handler_file)
+
+                try:
+                    with ButlerMDC.set_mdc(mdc):
+                        yield ctx
+                finally:
+                    # Ensure that the logs are stored in butler.
+                    logging.getLogger().removeHandler(log_handler_file)
+                    log_handler_file.close()
+                    if ctx.extra:
+                        with open(log_file, "a") as log_stream:
+                            ButlerLogRecords.write_streaming_extra(
+                                log_stream,
+                                ctx.extra.model_dump_json(exclude_unset=True, exclude_defaults=True),
+                            )
+                    if ctx.store:
+                        self._ingest_log_records(quantum, log_dataset_name, log_file)
+                    shutil.rmtree(tmpdir, ignore_errors=True)
 
             else:
-                log_handler_memory = ButlerLogRecordHandler(records)
+                log_handler_memory = ButlerLogRecordHandler()
                 logging.getLogger().addHandler(log_handler_memory)
 
                 try:
@@ -260,6 +261,7 @@ class LogCapture:
                     logging.getLogger().removeHandler(log_handler_memory)
                     if ctx.store:
                         self._store_log_records(quantum, log_dataset_name, log_handler_memory)
+                    log_handler_memory.records.clear()
 
         else:
             with ButlerMDC.set_mdc(mdc):
@@ -279,3 +281,41 @@ class LogCapture:
             ) from exc
 
         self.butler.put(log_handler.records, ref)
+
+    def _ingest_log_records(self, quantum: Quantum, dataset_type: str, filename: str) -> None:
+        # If we are logging to an external file we must always try to
+        # close it.
+        assert self.full_butler is not None, "Expected to have full butler for ingest"
+        ingested = False
+        try:
+            # DatasetRef has to be in the Quantum outputs, can lookup by name.
+            try:
+                [ref] = quantum.outputs[dataset_type]
+            except LookupError as exc:
+                raise InvalidQuantumError(
+                    f"Quantum outputs is missing log output dataset type {dataset_type};"
+                    " this could happen due to inconsistent options between QuantumGraph generation"
+                    " and execution"
+                ) from exc
+
+            # Need to ingest this file directly into butler.
+            dataset = FileDataset(path=filename, refs=ref)
+            try:
+                self.full_butler.ingest(dataset, transfer="move")
+                ingested = True
+            except NotImplementedError:
+                # Some datastores can't receive files (e.g. in-memory datastore
+                # when testing), we store empty list for those just to have a
+                # dataset. Alternative is to read the file as a
+                # ButlerLogRecords object and put it.
+                _LOG.info(
+                    "Log records could not be stored in this butler because the"
+                    " datastore can not ingest files, empty record list is stored instead."
+                )
+                records = ButlerLogRecords.from_records([])
+                self.full_butler.put(records, ref)
+        finally:
+            # remove file if it is not ingested
+            if not ingested:
+                with suppress(OSError):
+                    os.remove(filename)

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/mp_graph_executor.py

@@ -39,24 +39,20 @@ import sys
 import threading
 import time
 import uuid
-from contextlib import ExitStack
 from typing import Literal, cast
 
 import networkx
 
 from lsst.daf.butler import DataCoordinate, Quantum
 from lsst.daf.butler.cli.cliLog import CliLog
-from lsst.daf.butler.logging import ButlerLogRecords
 from lsst.utils.threads import disable_implicit_threading
 
 from ._status import InvalidQuantumError, RepeatableQuantumError
-from ._task_metadata import TaskMetadata
 from .execution_graph_fixup import ExecutionGraphFixup
 from .graph import QuantumGraph
 from .graph_walker import GraphWalker
-from .log_on_close import LogOnClose
 from .pipeline_graph import TaskNode
-from .quantum_graph import PredictedQuantumGraph, PredictedQuantumInfo, ProvenanceQuantumGraphWriter
+from .quantum_graph import PredictedQuantumGraph, PredictedQuantumInfo
 from .quantum_graph_executor import QuantumExecutor, QuantumGraphExecutor
 from .quantum_reports import ExecutionStatus, QuantumReport, Report
 
@@ -519,9 +515,7 @@ class MPGraphExecutor(QuantumGraphExecutor):
             start_method = "spawn"
         self._start_method = start_method
 
-    def execute(
-        self, graph: QuantumGraph | PredictedQuantumGraph, *, provenance_graph_file: str | None = None
-    ) -> None:
+    def execute(self, graph: QuantumGraph | PredictedQuantumGraph) -> None:
         # Docstring inherited from QuantumGraphExecutor.execute
         old_graph: QuantumGraph | None = None
         if isinstance(graph, QuantumGraph):
@@ -531,31 +525,14 @@ class MPGraphExecutor(QuantumGraphExecutor):
             new_graph = graph
         xgraph = self._make_xgraph(new_graph, old_graph)
         self._report = Report(qgraphSummary=new_graph._make_summary())
-        with ExitStack() as exit_stack:
-            provenance_writer: ProvenanceQuantumGraphWriter | None = None
-            if provenance_graph_file is not None:
-                if provenance_graph_file is not None and self._num_proc > 1:
-                    raise NotImplementedError(
-                        "Provenance writing is not implemented for multiprocess execution."
-                    )
-                provenance_writer = ProvenanceQuantumGraphWriter(
-                    provenance_graph_file,
-                    exit_stack=exit_stack,
-                    log_on_close=LogOnClose(_LOG.log),
-                    predicted=new_graph,
-                )
-            try:
-                if self._num_proc > 1:
-                    self._execute_quanta_mp(xgraph, self._report)
-                else:
-                    self._execute_quanta_in_process(xgraph, self._report, provenance_writer)
-            except Exception as exc:
-                self._report.set_exception(exc)
-                raise
-            if provenance_writer is not None:
-                provenance_writer.write_overall_inputs()
-                provenance_writer.write_packages()
-                provenance_writer.write_init_outputs(assume_existence=True)
+        try:
+            if self._num_proc > 1:
+                self._execute_quanta_mp(xgraph, self._report)
+            else:
+                self._execute_quanta_in_process(xgraph, self._report)
+        except Exception as exc:
+            self._report.set_exception(exc)
+            raise
 
     def _make_xgraph(
         self, new_graph: PredictedQuantumGraph, old_graph: QuantumGraph | None
@@ -599,9 +576,7 @@ class MPGraphExecutor(QuantumGraphExecutor):
                 raise MPGraphExecutorError("Updated execution graph has dependency cycle.")
         return xgraph
 
-    def _execute_quanta_in_process(
-        self, xgraph: networkx.DiGraph, report: Report, provenance_writer: ProvenanceQuantumGraphWriter | None
-    ) -> None:
+    def _execute_quanta_in_process(self, xgraph: networkx.DiGraph, report: Report) -> None:
         """Execute all Quanta in current process.
 
         Parameters
@@ -614,9 +589,6 @@ class MPGraphExecutor(QuantumGraphExecutor):
             `.quantum_graph.PredictedQuantumGraph.quantum_only_xgraph`.
         report : `Report`
             Object for reporting execution status.
-        provenance_writer : `.quantum_graph.ProvenanceQuantumGraphWriter` or \
-                `None`
-            Object for recording provenance.
         """
 
         def tiebreaker_sort_key(quantum_id: uuid.UUID) -> tuple:
@@ -634,19 +606,16 @@ class MPGraphExecutor(QuantumGraphExecutor):
 
                 _LOG.debug("Executing %s (%s@%s)", quantum_id, task_node.label, data_id)
                 fail_exit_code: int | None = None
-                task_metadata: TaskMetadata | None = None
-                task_logs = ButlerLogRecords([])
                 try:
                     # For some exception types we want to exit immediately with
                     # exception-specific exit code, but we still want to start
                     # debugger before exiting if debugging is enabled.
                     try:
-                        execution_result = self._quantum_executor.execute(
-                            task_node, quantum, quantum_id=quantum_id, log_records=task_logs
+                        _, quantum_report = self._quantum_executor.execute(
+                            task_node, quantum, quantum_id=quantum_id
                         )
-                        if execution_result.report:
-                            report.quantaReports.append(execution_result.report)
-                        task_metadata = execution_result.task_metadata
+                        if quantum_report:
+                            report.quantaReports.append(quantum_report)
                         success_count += 1
                         walker.finish(quantum_id)
                     except RepeatableQuantumError as exc:
@@ -732,11 +701,6 @@ class MPGraphExecutor(QuantumGraphExecutor):
                         )
                         failed_count += 1
 
-                if provenance_writer is not None:
-                    provenance_writer.write_quantum_provenance(
-                        quantum_id, metadata=task_metadata, logs=task_logs
-                    )
-
                 _LOG.info(
                     "Executed %d quanta successfully, %d failed and %d remain out of total %d quanta.",
                     success_count,

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/quantum_graph/_common.py

@@ -448,15 +448,14 @@ class BaseQuantumGraphWriter:
         uri: ResourcePathExpression,
         header: HeaderModel,
         pipeline_graph: PipelineGraph,
+        indices: dict[uuid.UUID, int],
         *,
         address_filename: str,
+        compressor: Compressor,
         cdict_data: bytes | None = None,
-        zstd_level: int = 10,
     ) -> Iterator[Self]:
         uri = ResourcePath(uri)
-        address_writer = AddressWriter()
-        cdict = zstandard.ZstdCompressionDict(cdict_data) if cdict_data is not None else None
-        compressor = zstandard.ZstdCompressor(level=zstd_level, dict_data=cdict)
+        address_writer = AddressWriter(indices)
         with uri.open(mode="wb") as stream:
             with zipfile.ZipFile(stream, mode="w", compression=zipfile.ZIP_STORED) as zf:
                 self = cls(zf, compressor, address_writer, header.int_size)

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/quantum_graph/_multiblock.py

@@ -205,6 +205,13 @@ class AddressRow:
 class AddressWriter:
     """A helper object for writing address files for multi-block files."""
 
+    indices: dict[uuid.UUID, int] = dataclasses.field(default_factory=dict)
+    """Mapping from UUID to internal integer ID.
+
+    The internal integer ID must always correspond to the index into the
+    sorted list of all UUIDs, but this `dict` need not be sorted itself.
+    """
+
     addresses: list[dict[uuid.UUID, Address]] = dataclasses.field(default_factory=list)
     """Addresses to store with each UUID.
 
@@ -222,15 +229,18 @@ class AddressWriter:
         int_size : `int`
             Number of bytes to use for all integers.
         """
-        indices: set[uuid.UUID] = set()
-        for address_map in self.addresses:
-            indices.update(address_map.keys())
+        for n, address_map in enumerate(self.addresses):
+            if not self.indices.keys() >= address_map.keys():
+                raise AssertionError(
+                    f"Logic bug in quantum graph I/O: address map {n} of {len(self.addresses)} has IDs "
+                    f"{address_map.keys() - self.indices.keys()} not in the index map."
+                )
         stream.write(int_size.to_bytes(1))
-        stream.write(len(indices).to_bytes(int_size))
+        stream.write(len(self.indices).to_bytes(int_size))
         stream.write(len(self.addresses).to_bytes(int_size))
         empty_address = Address()
-        for n, key in enumerate(sorted(indices, key=attrgetter("int"))):
-            row = AddressRow(key, n, [m.get(key, empty_address) for m in self.addresses])
+        for key in sorted(self.indices.keys(), key=attrgetter("int")):
+            row = AddressRow(key, self.indices[key], [m.get(key, empty_address) for m in self.addresses])
             _LOG.debug("Wrote address %s.", row)
             row.write(stream, int_size)
 

{lsst_pipe_base-30.0.0 → lsst_pipe_base-30.0.0rc2}/python/lsst/pipe/base/quantum_graph/_predicted.py

@@ -66,7 +66,6 @@ from lsst.daf.butler import (
     DimensionDataExtractor,
     DimensionGroup,
     DimensionRecordSetDeserializer,
-    DimensionUniverse,
     LimitedButler,
     Quantum,
     QuantumBackedButler,
@@ -199,10 +198,10 @@ class _PredictedThinGraphModelV0(pydantic.BaseModel):
     def _upgraded(self, address_rows: Mapping[uuid.UUID, AddressRow]) -> PredictedThinGraphModel:
         """Convert to the v1+ model."""
         uuid_by_index = {v.index: k for k, v in address_rows.items()}
-        return PredictedThinGraphModel.model_construct(
+        return PredictedThinGraphModel(
             quanta={
                 task_label: [
-                    PredictedThinQuantumModel.model_construct(
+                    PredictedThinQuantumModel(
                         quantum_id=uuid_by_index[q.quantum_index], data_coordinate=q.data_coordinate
                     )
                     for q in quanta
@@ -878,49 +877,6 @@ class PredictedQuantumGraph(BaseQuantumGraph):
             page_size=page_size,
         ).assemble()
 
-    @classmethod
-    def make_empty(
-        cls,
-        universe: DimensionUniverse,
-        *,
-        output_run: str,
-        inputs: Iterable[str] = (),
-        output: str | None = None,
-        add_packages: bool = True,
-    ) -> PredictedQuantumGraph:
-        """Make an empty quantum graph with no tasks.
-
-        Parameters
-        ----------
-        universe : `lsst.daf.butler.DimensionUniverse`
-            Definitions for all butler dimensions.
-        output_run : `str`
-            Output run collection.
-        inputs : `~collections.abc.Iterable` [`str`], optional
-            Iterable of input collection names.
-        output : `str` or `None`, optional
-            Output chained collection.
-        add_packages : `bool`, optional
-            Whether to add the special init quantum that writes the 'packages'
-            dataset.  The default (`True`) is consistent with
-            `~..quantum_graph_builder.QuantumGraphBuilder` behavior when there
-            are no regular quanta generated.
-
-        Returns
-        -------
-        quantum_graph : `PredictedQuantumGraph`
-            An empty quantum graph.
-        """
-        return cls(
-            PredictedQuantumGraphComponents.make_empty(
-                universe,
-                output_run=output_run,
-                inputs=inputs,
-                output=output,
-                add_packages=add_packages,
-            )
-        )
-
     @property
     def quanta_by_task(self) -> Mapping[str, Mapping[DataCoordinate, uuid.UUID]]:
         """A nested mapping of all quanta, keyed first by task name and then by
@@ -1585,63 +1541,6 @@ class PredictedQuantumGraphComponents:
     This does not include special "init" quanta.
     """
 
-    @classmethod
-    def make_empty(
-        cls,
-        universe: DimensionUniverse,
-        *,
-        output_run: str,
-        inputs: Iterable[str] = (),
-        output: str | None = None,
-        add_packages: bool = True,
-    ) -> PredictedQuantumGraphComponents:
-        """Make components for an empty quantum graph with no tasks.
-
-        Parameters
-        ----------
-        universe : `lsst.daf.butler.DimensionUniverse`
-            Definitions for all butler dimensions.
-        output_run : `str`
-            Output run collection.
-        inputs : `~collections.abc.Iterable` [`str`], optional
-            Iterable of input collection names.
-        output : `str` or `None`, optional
-            Output chained collection.
-        add_packages : `bool`, optional
-            Whether to add the special init quantum that writes the 'packages'
-            dataset.  The default (`True`) is consistent with
-            `~..quantum_graph_builder.QuantumGraphBuilder` behavior when there
-            are no regular quanta generated.
-
-        Returns
-        -------
-        components : `PredictedQuantumGraphComponents`
-            Components that can be used to build or write an empty quantum
-            graph.
-        """
-        components = cls(pipeline_graph=PipelineGraph(universe=universe))
-        components.header.inputs = list(inputs)
-        components.header.output_run = output_run
-        components.header.output = output
-        if add_packages:
-            components.init_quanta.root = [
-                PredictedQuantumDatasetsModel.model_construct(
-                    quantum_id=generate_uuidv7(),
-                    task_label="",
-                    outputs={
-                        acc.PACKAGES_INIT_OUTPUT_NAME: [
-                            PredictedDatasetModel(
-                                dataset_id=generate_uuidv7(),
-                                dataset_type_name=acc.PACKAGES_INIT_OUTPUT_NAME,
-                                data_coordinate=[],
-                                run=output_run,
-                            )
-                        ]
-                    },
-                )
-            ]
-        return components
-
     def make_dataset_ref(self, predicted: PredictedDatasetModel) -> DatasetRef:
         """Make a `lsst.daf.butler.DatasetRef` from information in the
         predicted quantum graph.
@@ -1894,6 +1793,7 @@ class PredictedQuantumGraphComponents:
                     f"Unsupported extension {ext!r} for quantum graph; "
                     "expected '.qg' (or '.qgraph' to force the old format)."
                 )
+        cdict: zstandard.ZstdCompressionDict | None = None
         cdict_data: bytes | None = None
         quantum_datasets_json: dict[uuid.UUID, bytes] = {}
         if len(self.quantum_datasets) < zstd_dict_n_inputs:
@@ -1907,20 +1807,26 @@ class PredictedQuantumGraphComponents:
                 for quantum_model in itertools.islice(self.quantum_datasets.values(), zstd_dict_n_inputs)
             }
             try:
-                cdict_data = zstandard.train_dictionary(
+                cdict = zstandard.train_dictionary(
                     zstd_dict_size,
                     list(quantum_datasets_json.values()),
                     level=zstd_level,
-                ).as_bytes()
+                )
             except zstandard.ZstdError as err:
                 warnings.warn(f"Not using a compression dictionary: {err}.")
+                cdict = None
+            else:
+                cdict_data = cdict.as_bytes()
+        compressor = zstandard.ZstdCompressor(level=zstd_level, dict_data=cdict)
+        indices = {quantum_id: n for n, quantum_id in enumerate(sorted(self.quantum_datasets.keys()))}
         with BaseQuantumGraphWriter.open(
             uri,
             header=self.header,
             pipeline_graph=self.pipeline_graph,
+            indices=indices,
             address_filename="quanta",
+            compressor=compressor,
             cdict_data=cdict_data,
-            zstd_level=zstd_level,
         ) as writer:
             writer.write_single_model("thin_graph", self.thin_graph)
             if self.dimension_data is None: