lsst-pipe-base 30.0.0rc1__tar.gz → 30.0.0rc3__tar.gz

{lsst_pipe_base-30.0.0rc1/python/lsst_pipe_base.egg-info → lsst_pipe_base-30.0.0rc3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-pipe-base
-Version: 30.0.0rc1
+Version: 30.0.0rc3
 Summary: Pipeline infrastructure for the Rubin Science Pipelines.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License-Expression: BSD-3-Clause OR GPL-3.0-or-later

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/doc/lsst.pipe.base/CHANGES.rst

@@ -1,3 +1,69 @@
+lsst-pipe-base v30.0.0 (2026-01-16)
+===================================
+
+New Features
+------------
+
+- Added support for healpix (and other non-database dimensions) in quantum graph builder. (`DM-51176 <https://rubinobs.atlassian.net/browse/DM-51176>`_)
+- Added filtering out dataset refs that the destination butler already knows in ``transfer_from_graph`` as well as dividing the transfer into smaller chunks to speed up restarts. (`DM-51273 <https://rubinobs.atlassian.net/browse/DM-51273>`_)
+- Added handling of ``PrerequisiteInput`` in ``QuantaAdjuster``, with a corresponding unit test. (`DM-51509 <https://rubinobs.atlassian.net/browse/DM-51509>`_)
+- Added ``PredictedQuantumGraph``, a replacement for the ``QuantumGraph`` class with more efficient I/O (via a new file format and more partial-read flexibility).
+
+  The new ``PredictedQuantumGraph`` is now the default in most tooling, and the new format can be opted into via the ``.qg`` (instead of ``.qgraph``) file extension.
+  New files can be read with the old class and vice versa.
+
+  The ``QuantumGraph`` class will eventually be deprecated along with much of the current provenance reporting tooling, but only when the new provenance ecosystem is fully in place. (`DM-51850 <https://rubinobs.atlassian.net/browse/DM-51850>`_)
+- Added a ``rename`` dict in ``ImportIR`` to support renaming task labels, with corresponding unit tests. (`DM-52168 <https://rubinobs.atlassian.net/browse/DM-52168>`_)
+- Added the new ``ProvenanceQuantumGraph`` class and the ``aggregate-graph`` tool (a replacement for ``transfer-from-graph``) that writes it at the end of batch runs. (`DM-52360 <https://rubinobs.atlassian.net/browse/DM-52360>`_)
+- Improved provenance tracking for failed quanta and retries.
+
+  By storing extra information in the log datasets written during execution,
+  we can record caught exceptions, track which other quanta have
+  already executed in the same process, and keep track of previous attempts to
+  run the same quantum. (`DM-53019 <https://rubinobs.atlassian.net/browse/DM-53019>`_)
+- Added provenance writing support to ``MPGraphExecutor`` and ``SeparablePipelineExecutor``. (`DM-53622 <https://rubinobs.atlassian.net/browse/DM-53622>`_)
+
+
+API Changes
+-----------
+
+- Moved pipeline executors and their support code here, from ``lsst.ctrl.mpexec``.
+
+  This included minor API changes for ``SingleQuantumExecutor`` as well: consistent snake-case naming, keyword-only arguments for construction, and a switch to private instance attributes. (`DM-48980 <https://rubinobs.atlassian.net/browse/DM-48980>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed ``transfer-from-graph`` to update chain when asked and output run collection exists even if didn't transfer any datasets. (`DM-51821 <https://rubinobs.atlassian.net/browse/DM-51821>`_)
+- Fixed bug in ``transfer_from_graph`` where the input collections were not flattened before adding to a new output chain. (`DM-52004 <https://rubinobs.atlassian.net/browse/DM-52004>`_)
+- Fixed bug where the log's ``MDC.RUN`` was the empty string when using a quantum-backed butler. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
+- Fixed a bug that caused ``PipelineGraph`` objects to be marked as unresolved when loaded from disk. (`DM-52787 <https://rubinobs.atlassian.net/browse/DM-52787>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- The ``Instrument.configPaths`` property can now refer to ``lsst.resources.ResourcePath`` URIs as well as strings (paths or URI strings). (`DM-33226 <https://rubinobs.atlassian.net/browse/DM-33226>`_)
+- Updated pipe_base code to use the constants defined in ``automatic_connection_constants.py``. (`DM-52676 <https://rubinobs.atlassian.net/browse/DM-52676>`_)
+- Uses UUIDs instead of internal integer IDs in new quantum graph storage.
+
+  This includes backwards compatibility read support for predicted quantum graphs, but not provenance quantum graphs, because those are still experimental anyway.
+
+  This increases the size of the files by ~6%, but it simplifies the codebase and will make consolidating multiple small provenance quantum graphs (as we
+  currently anticipate doing for prompt processing) much more efficient. (`DM-53174 <https://rubinobs.atlassian.net/browse/DM-53174>`_)
+- Used context managers to ensure that database resources are freed. (`DM-53370 <https://rubinobs.atlassian.net/browse/DM-53370>`_)
+- Added more logging for the later steps of QG building. (`DM-53636 <https://rubinobs.atlassian.net/browse/DM-53636>`_)
+
+
+An API Removal or Deprecation
+-----------------------------
+
+- Removes the ``buildExecutionButler`` function and all supporting code.
+
+  Execution butlers (read-only SQLite databases used for batch execution) have been fully superseded by ``lsst.daf.butler.QuantumBackedButler``. (`DM-52044 <https://rubinobs.atlassian.net/browse/DM-52044>`_)
+
+
 lsst-pipe-base v29.1.0 (2025-06-13)
 ===================================
 

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/python/lsst/pipe/base/_instrument.py

@@ -31,7 +31,6 @@ __all__ = ("Instrument",)
 
 import contextlib
 import datetime
-import os.path
 from abc import ABCMeta, abstractmethod
 from collections.abc import Sequence
 from typing import TYPE_CHECKING, Any, Self, cast, final
@@ -39,6 +38,7 @@ from typing import TYPE_CHECKING, Any, Self, cast, final
 from lsst.daf.butler import DataCoordinate, DataId, DimensionPacker, DimensionRecord, Formatter
 from lsst.daf.butler.registry import DataIdError
 from lsst.pex.config import Config, RegistryField
+from lsst.resources import ResourcePath, ResourcePathExpression
 from lsst.utils import doImportType
 from lsst.utils.introspection import get_full_type_name
 
@@ -65,7 +65,7 @@ class Instrument(metaclass=ABCMeta):
     the base class.
     """
 
-    configPaths: Sequence[str] = ()
+    configPaths: Sequence[ResourcePathExpression] = ()
     """Paths to config files to read for specific Tasks.
 
     The paths in this list should contain files of the form `task.py`, for
@@ -366,9 +366,10 @@ class Instrument(metaclass=ABCMeta):
             Config instance to which overrides should be applied.
         """
         for root in self.configPaths:
-            path = os.path.join(root, f"{name}.py")
-            if os.path.exists(path):
-                config.load(path)
+            resource = ResourcePath(root, forceDirectory=True, forceAbsolute=True)
+            uri = resource.join(f"{name}.py", forceDirectory=False)
+            if uri.exists():
+                config.load(uri)
 
     @staticmethod
     def formatCollectionTimestamp(timestamp: str | datetime.datetime) -> str:

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/python/lsst/pipe/base/log_capture.py

@@ -31,17 +31,15 @@ __all__ = ["LogCapture"]
 
 import dataclasses
 import logging
-import os
-import shutil
-import tempfile
 import uuid
 from collections.abc import Iterator
-from contextlib import contextmanager, suppress
+from contextlib import contextmanager
 from logging import FileHandler
 
 import pydantic
 
-from lsst.daf.butler import Butler, FileDataset, LimitedButler, Quantum
+from lsst.daf.butler import Butler, LimitedButler, Quantum
+from lsst.daf.butler._rubin.temporary_for_ingest import TemporaryForIngest
 from lsst.daf.butler.logging import (
     ButlerLogRecord,
     ButlerLogRecordHandler,
@@ -165,7 +163,9 @@ class LogCapture:
         return cls(butler, butler)
 
     @contextmanager
-    def capture_logging(self, task_node: TaskNode, /, quantum: Quantum) -> Iterator[_LogCaptureContext]:
+    def capture_logging(
+        self, task_node: TaskNode, /, quantum: Quantum, records: ButlerLogRecords | None = None
+    ) -> Iterator[_LogCaptureContext]:
         """Configure logging system to capture logs for execution of this task.
 
         Parameters
@@ -174,6 +174,9 @@ class LogCapture:
             The task definition.
         quantum : `~lsst.daf.butler.Quantum`
             Single Quantum instance.
+        records : `lsst.daf.butler.logging.ButlerLogRecords`, optional
+            Log record container to append to and save.  If provided, streaming
+            mode is disabled (since we'll be saving logs in memory anyway).
 
         Notes
         -----
@@ -205,44 +208,40 @@ class LogCapture:
 
         # Add a handler to the root logger to capture execution log output.
         if log_dataset_name is not None:
+            try:
+                [ref] = quantum.outputs[log_dataset_name]
+            except LookupError as exc:
+                raise InvalidQuantumError(
+                    f"Quantum outputs is missing log output dataset type {log_dataset_name};"
+                    " this could happen due to inconsistent options between QuantumGraph generation"
+                    " and execution"
+                ) from exc
             # Either accumulate into ButlerLogRecords or stream JSON records to
             # file and ingest that (ingest is possible only with full butler).
-            if self.stream_json_logs and self.full_butler is not None:
-                # Create the log file in a temporary directory rather than
-                # creating a temporary file. This is necessary because
-                # temporary files are created with restrictive permissions
-                # and during file ingest these permissions persist in the
-                # datastore. Using a temp directory allows us to create
-                # a file with umask default permissions.
-                tmpdir = tempfile.mkdtemp(prefix="butler-temp-logs-")
-
-                # Construct a file to receive the log records and "touch" it.
-                log_file = os.path.join(tmpdir, f"butler-log-{task_node.label}.json")
-                with open(log_file, "w"):
-                    pass
-                log_handler_file = FileHandler(log_file)
-                log_handler_file.setFormatter(JsonLogFormatter())
-                logging.getLogger().addHandler(log_handler_file)
-
-                try:
-                    with ButlerMDC.set_mdc(mdc):
-                        yield ctx
-                finally:
-                    # Ensure that the logs are stored in butler.
-                    logging.getLogger().removeHandler(log_handler_file)
-                    log_handler_file.close()
-                    if ctx.extra:
-                        with open(log_file, "a") as log_stream:
-                            ButlerLogRecords.write_streaming_extra(
-                                log_stream,
-                                ctx.extra.model_dump_json(exclude_unset=True, exclude_defaults=True),
-                            )
-                    if ctx.store:
-                        self._ingest_log_records(quantum, log_dataset_name, log_file)
-                    shutil.rmtree(tmpdir, ignore_errors=True)
+            if self.stream_json_logs and self.full_butler is not None and records is None:
+                with TemporaryForIngest(self.full_butler, ref) as temporary:
+                    log_handler_file = FileHandler(temporary.ospath)
+                    log_handler_file.setFormatter(JsonLogFormatter())
+                    logging.getLogger().addHandler(log_handler_file)
+
+                    try:
+                        with ButlerMDC.set_mdc(mdc):
+                            yield ctx
+                    finally:
+                        # Ensure that the logs are stored in butler.
+                        logging.getLogger().removeHandler(log_handler_file)
+                        log_handler_file.close()
+                        if ctx.extra:
+                            with open(temporary.ospath, "a") as log_stream:
+                                ButlerLogRecords.write_streaming_extra(
+                                    log_stream,
+                                    ctx.extra.model_dump_json(exclude_unset=True, exclude_defaults=True),
+                                )
+                        if ctx.store:
+                            temporary.ingest()
 
             else:
-                log_handler_memory = ButlerLogRecordHandler()
+                log_handler_memory = ButlerLogRecordHandler(records)
                 logging.getLogger().addHandler(log_handler_memory)
 
                 try:
@@ -261,7 +260,6 @@ class LogCapture:
                     logging.getLogger().removeHandler(log_handler_memory)
                     if ctx.store:
                         self._store_log_records(quantum, log_dataset_name, log_handler_memory)
-                    log_handler_memory.records.clear()
 
         else:
             with ButlerMDC.set_mdc(mdc):
@@ -281,41 +279,3 @@ class LogCapture:
             ) from exc
 
         self.butler.put(log_handler.records, ref)
-
-    def _ingest_log_records(self, quantum: Quantum, dataset_type: str, filename: str) -> None:
-        # If we are logging to an external file we must always try to
-        # close it.
-        assert self.full_butler is not None, "Expected to have full butler for ingest"
-        ingested = False
-        try:
-            # DatasetRef has to be in the Quantum outputs, can lookup by name.
-            try:
-                [ref] = quantum.outputs[dataset_type]
-            except LookupError as exc:
-                raise InvalidQuantumError(
-                    f"Quantum outputs is missing log output dataset type {dataset_type};"
-                    " this could happen due to inconsistent options between QuantumGraph generation"
-                    " and execution"
-                ) from exc
-
-            # Need to ingest this file directly into butler.
-            dataset = FileDataset(path=filename, refs=ref)
-            try:
-                self.full_butler.ingest(dataset, transfer="move")
-                ingested = True
-            except NotImplementedError:
-                # Some datastores can't receive files (e.g. in-memory datastore
-                # when testing), we store empty list for those just to have a
-                # dataset. Alternative is to read the file as a
-                # ButlerLogRecords object and put it.
-                _LOG.info(
-                    "Log records could not be stored in this butler because the"
-                    " datastore can not ingest files, empty record list is stored instead."
-                )
-                records = ButlerLogRecords.from_records([])
-                self.full_butler.put(records, ref)
-        finally:
-            # remove file if it is not ingested
-            if not ingested:
-                with suppress(OSError):
-                    os.remove(filename)

lsst_pipe_base-30.0.0rc3/python/lsst/pipe/base/log_on_close.py

@@ -0,0 +1,79 @@
+# This file is part of pipe_base.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (http://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This software is dual licensed under the GNU General Public License and also
+# under a 3-clause BSD license. Recipients may choose which of these licenses
+# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
+# respectively.  If you choose the GPL option then the following text applies
+# (but note that there is still no warranty even if you opt for BSD instead):
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import annotations
+
+__all__ = ("LogOnClose",)
+
+from collections.abc import Callable, Iterator
+from contextlib import AbstractContextManager, contextmanager
+from typing import TypeVar
+
+from lsst.utils.logging import VERBOSE
+
+_T = TypeVar("_T")
+
+
+class LogOnClose:
+    """A factory for context manager wrappers that emit a log message when
+    they are closed.
+
+    Parameters
+    ----------
+    log_func : `~collections.abc.Callable` [ `int`, `str` ]
+        Callable that takes an integer log level and a string message and emits
+        a log message.  Note that placeholder formatting is not supported.
+    """
+
+    def __init__(self, log_func: Callable[[int, str], None]):
+        self.log_func = log_func
+
+    def wrap(
+        self,
+        cm: AbstractContextManager[_T],
+        msg: str,
+        level: int = VERBOSE,
+    ) -> AbstractContextManager[_T]:
+        """Wrap a context manager to log when it is exited.
+
+        Parameters
+        ----------
+        cm : `contextlib.AbstractContextManager`
+            Context manager to wrap.
+        msg : `str`
+            Log message.
+        level : `int`, optional
+            Log level.
+        """
+
+        @contextmanager
+        def wrapper() -> Iterator[_T]:
+            with cm as result:
+                yield result
+                self.log_func(level, msg)
+
+        return wrapper()

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/python/lsst/pipe/base/mp_graph_executor.py

@@ -39,20 +39,24 @@ import sys
 import threading
 import time
 import uuid
+from contextlib import ExitStack
 from typing import Literal, cast
 
 import networkx
 
 from lsst.daf.butler import DataCoordinate, Quantum
 from lsst.daf.butler.cli.cliLog import CliLog
+from lsst.daf.butler.logging import ButlerLogRecords
 from lsst.utils.threads import disable_implicit_threading
 
 from ._status import InvalidQuantumError, RepeatableQuantumError
+from ._task_metadata import TaskMetadata
 from .execution_graph_fixup import ExecutionGraphFixup
 from .graph import QuantumGraph
 from .graph_walker import GraphWalker
+from .log_on_close import LogOnClose
 from .pipeline_graph import TaskNode
-from .quantum_graph import PredictedQuantumGraph, PredictedQuantumInfo
+from .quantum_graph import PredictedQuantumGraph, PredictedQuantumInfo, ProvenanceQuantumGraphWriter
 from .quantum_graph_executor import QuantumExecutor, QuantumGraphExecutor
 from .quantum_reports import ExecutionStatus, QuantumReport, Report
 
@@ -515,7 +519,9 @@ class MPGraphExecutor(QuantumGraphExecutor):
             start_method = "spawn"
         self._start_method = start_method
 
-    def execute(self, graph: QuantumGraph | PredictedQuantumGraph) -> None:
+    def execute(
+        self, graph: QuantumGraph | PredictedQuantumGraph, *, provenance_graph_file: str | None = None
+    ) -> None:
         # Docstring inherited from QuantumGraphExecutor.execute
         old_graph: QuantumGraph | None = None
         if isinstance(graph, QuantumGraph):
@@ -525,14 +531,31 @@ class MPGraphExecutor(QuantumGraphExecutor):
             new_graph = graph
         xgraph = self._make_xgraph(new_graph, old_graph)
         self._report = Report(qgraphSummary=new_graph._make_summary())
-        try:
-            if self._num_proc > 1:
-                self._execute_quanta_mp(xgraph, self._report)
-            else:
-                self._execute_quanta_in_process(xgraph, self._report)
-        except Exception as exc:
-            self._report.set_exception(exc)
-            raise
+        with ExitStack() as exit_stack:
+            provenance_writer: ProvenanceQuantumGraphWriter | None = None
+            if provenance_graph_file is not None:
+                if provenance_graph_file is not None and self._num_proc > 1:
+                    raise NotImplementedError(
+                        "Provenance writing is not implemented for multiprocess execution."
+                    )
+                provenance_writer = ProvenanceQuantumGraphWriter(
+                    provenance_graph_file,
+                    exit_stack=exit_stack,
+                    log_on_close=LogOnClose(_LOG.log),
+                    predicted=new_graph,
+                )
+            try:
+                if self._num_proc > 1:
+                    self._execute_quanta_mp(xgraph, self._report)
+                else:
+                    self._execute_quanta_in_process(xgraph, self._report, provenance_writer)
+            except Exception as exc:
+                self._report.set_exception(exc)
+                raise
+            if provenance_writer is not None:
+                provenance_writer.write_overall_inputs()
+                provenance_writer.write_packages()
+                provenance_writer.write_init_outputs(assume_existence=True)
 
     def _make_xgraph(
         self, new_graph: PredictedQuantumGraph, old_graph: QuantumGraph | None
@@ -576,7 +599,9 @@ class MPGraphExecutor(QuantumGraphExecutor):
                 raise MPGraphExecutorError("Updated execution graph has dependency cycle.")
         return xgraph
 
-    def _execute_quanta_in_process(self, xgraph: networkx.DiGraph, report: Report) -> None:
+    def _execute_quanta_in_process(
+        self, xgraph: networkx.DiGraph, report: Report, provenance_writer: ProvenanceQuantumGraphWriter | None
+    ) -> None:
         """Execute all Quanta in current process.
 
         Parameters
@@ -589,6 +614,9 @@ class MPGraphExecutor(QuantumGraphExecutor):
             `.quantum_graph.PredictedQuantumGraph.quantum_only_xgraph`.
         report : `Report`
             Object for reporting execution status.
+        provenance_writer : `.quantum_graph.ProvenanceQuantumGraphWriter` or \
+                `None`
+            Object for recording provenance.
         """
 
         def tiebreaker_sort_key(quantum_id: uuid.UUID) -> tuple:
@@ -606,16 +634,19 @@ class MPGraphExecutor(QuantumGraphExecutor):
 
                 _LOG.debug("Executing %s (%s@%s)", quantum_id, task_node.label, data_id)
                 fail_exit_code: int | None = None
+                task_metadata: TaskMetadata | None = None
+                task_logs = ButlerLogRecords([])
                 try:
                     # For some exception types we want to exit immediately with
                     # exception-specific exit code, but we still want to start
                     # debugger before exiting if debugging is enabled.
                     try:
-                        _, quantum_report = self._quantum_executor.execute(
-                            task_node, quantum, quantum_id=quantum_id
+                        execution_result = self._quantum_executor.execute(
+                            task_node, quantum, quantum_id=quantum_id, log_records=task_logs
                         )
-                        if quantum_report:
-                            report.quantaReports.append(quantum_report)
+                        if execution_result.report:
+                            report.quantaReports.append(execution_result.report)
+                        task_metadata = execution_result.task_metadata
                         success_count += 1
                         walker.finish(quantum_id)
                     except RepeatableQuantumError as exc:
@@ -701,6 +732,11 @@ class MPGraphExecutor(QuantumGraphExecutor):
                         )
                         failed_count += 1
 
+                if provenance_writer is not None:
+                    provenance_writer.write_quantum_provenance(
+                        quantum_id, metadata=task_metadata, logs=task_logs
+                    )
+
                 _LOG.info(
                     "Executed %d quanta successfully, %d failed and %d remain out of total %d quanta.",
                     success_count,

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/python/lsst/pipe/base/quantum_graph/_common.py

@@ -448,14 +448,15 @@ class BaseQuantumGraphWriter:
         uri: ResourcePathExpression,
         header: HeaderModel,
         pipeline_graph: PipelineGraph,
-        indices: dict[uuid.UUID, int],
         *,
         address_filename: str,
-        compressor: Compressor,
         cdict_data: bytes | None = None,
+        zstd_level: int = 10,
     ) -> Iterator[Self]:
         uri = ResourcePath(uri)
-        address_writer = AddressWriter(indices)
+        address_writer = AddressWriter()
+        cdict = zstandard.ZstdCompressionDict(cdict_data) if cdict_data is not None else None
+        compressor = zstandard.ZstdCompressor(level=zstd_level, dict_data=cdict)
         with uri.open(mode="wb") as stream:
             with zipfile.ZipFile(stream, mode="w", compression=zipfile.ZIP_STORED) as zf:
                 self = cls(zf, compressor, address_writer, header.int_size)

{lsst_pipe_base-30.0.0rc1 → lsst_pipe_base-30.0.0rc3}/python/lsst/pipe/base/quantum_graph/_multiblock.py

@@ -205,13 +205,6 @@ class AddressRow:
 class AddressWriter:
     """A helper object for writing address files for multi-block files."""
 
-    indices: dict[uuid.UUID, int] = dataclasses.field(default_factory=dict)
-    """Mapping from UUID to internal integer ID.
-
-    The internal integer ID must always correspond to the index into the
-    sorted list of all UUIDs, but this `dict` need not be sorted itself.
-    """
-
     addresses: list[dict[uuid.UUID, Address]] = dataclasses.field(default_factory=list)
     """Addresses to store with each UUID.
 
@@ -229,18 +222,15 @@ class AddressWriter:
         int_size : `int`
             Number of bytes to use for all integers.
         """
-        for n, address_map in enumerate(self.addresses):
-            if not self.indices.keys() >= address_map.keys():
-                raise AssertionError(
-                    f"Logic bug in quantum graph I/O: address map {n} of {len(self.addresses)} has IDs "
-                    f"{address_map.keys() - self.indices.keys()} not in the index map."
-                )
+        indices: set[uuid.UUID] = set()
+        for address_map in self.addresses:
+            indices.update(address_map.keys())
         stream.write(int_size.to_bytes(1))
-        stream.write(len(self.indices).to_bytes(int_size))
+        stream.write(len(indices).to_bytes(int_size))
         stream.write(len(self.addresses).to_bytes(int_size))
         empty_address = Address()
-        for key in sorted(self.indices.keys(), key=attrgetter("int")):
-            row = AddressRow(key, self.indices[key], [m.get(key, empty_address) for m in self.addresses])
+        for n, key in enumerate(sorted(indices, key=attrgetter("int"))):
+            row = AddressRow(key, n, [m.get(key, empty_address) for m in self.addresses])
             _LOG.debug("Wrote address %s.", row)
             row.write(stream, int_size)