lsst-daf-butler 29.2025.1200__tar.gz → 29.2025.1300__tar.gz

{lsst_daf_butler-29.2025.1200/python/lsst_daf_butler.egg-info → lsst_daf_butler-29.2025.1300}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-daf-butler
-Version: 29.2025.1200
+Version: 29.2025.1300
 Summary: An abstraction layer for reading and writing astronomical data to datastores.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License: BSD 3-Clause License

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/doc/lsst.daf.butler/CHANGES.rst

@@ -1,5 +1,73 @@
-Butler v28.0.0 2024-11-20
-=========================
+Butler v29.0.0 (2025-03-25)
+===========================
+
+New Features
+------------
+
+- * Added new class ``DatasetProvenance`` for tracking the provenance of an individual dataset.
+  * Modified ``Butler.put()`` to accept an optional ``DatasetProvenance``.
+  * Added ``add_provenance`` methods to ``FormatterV2`` and ``StorageClassDelegate``.
+    These methods will now be called with the provenance object during ``Butler.put()`` to allow the in-memory dataset to be updated prior to writing. (`DM-35396 <https://rubinobs.atlassian.net/browse/DM-35396>`_)
+- * Added ``Butler.retrieve_artifacts_zip`` and ``QuantumBackedButler.retrieve_artifacts_zip`` methods to retrieve the dataset artifacts and store them into a zip file.
+  * Added ``Butler.ingest_zip`` to ingest the contents of a Zip file.
+  * Added ``SerializedDatasetRefContainerV1`` class to allow a collection of ``DatasetRef`` to be serialized efficiently.
+    JSON serializations made using this class will be supported.
+  * Added ``--zip`` parameter to ``butler retrieve-artifacts``.
+  * Changed ``Butler.retrieveArtifacts`` to always write a JSON index file describing where the artifacts came from.
+  * Added a ``butler ingest-zip`` command-line tool for ingesting zip files created by ``butler retrieve-artifacts``. (`DM-46776 <https://rubinobs.atlassian.net/browse/DM-46776>`_)
+- The ``DAF_BUTLER_PLUGINS`` environment variable should no longer be set if packages use ``pip install`` and have been upgraded to use entry points.
+  Butler can now read the subcommands from ``pipe_base`` and ``daf_butler_migrate`` automatically.
+  Setting the environment variable for these packages will result in an error. (`DM-47143 <https://rubinobs.atlassian.net/browse/DM-47143>`_)
+- Added two new APIs for handling Butler dataset URIs.
+  ``Butler.parse_dataset_uri`` parses a URI and returns the butler repository label and associated UUID.
+  ``Butler.get_dataset_from_uri`` will parse a URI and attempt to retrieve the ``DatasetRef``.
+  URIs should be in the form of IVOA identifiers as described in `DMTN-302 <https://dmtn-302.lsst.io>`_.
+  Deprecated ``butler://`` URIs are still supported but should not be used in new systems. (`DM-47325 <https://rubinobs.atlassian.net/browse/DM-47325>`_)
+- Added a ``--chains NO-CHILDREN`` mode to the ``butler query-collections`` CLI,
+  which returns results without recursing into ``CHAINED`` collections. (`DM-47768 <https://rubinobs.atlassian.net/browse/DM-47768>`_)
+- Added ``lsst.daf.butler.formatters.parquet.add_pandas_index_to_astropy()`` function which stores special metadata that will be used to create a pandas DataFrame index if the table is read as a ``DataFrame``. (`DM-48141 <https://rubinobs.atlassian.net/browse/DM-48141>`_)
+- Modified the Obscore ``RecordFactory`` to support per-universe subclass discovery using entry points.
+
+  * Added ``RecordFactory.get_record_type_from_universe`` to obtain the correct factory class.
+  * Renamed ``ExposureRegionFactory`` to ``DerivedRegionFactory`` to make it clearer that this class is not solely used for exposures but the usage can change with universe.
+  * Added ``RecordFactory.region_dimension`` to return the dimension that would be needed to obtain a region for this universe. (`DM-48282 <https://rubinobs.atlassian.net/browse/DM-48282>`_)
+- * Added new methods to ``DatasetProvenance`` for serializing provenance to a flat dictionary and recovering provenance from that dictionary.
+  * Modifed ``ParquetFormatter`` to write provenance metadata to Astropy tables. (`DM-48869 <https://rubinobs.atlassian.net/browse/DM-48869>`_)
+
+
+API Changes
+-----------
+
+- Added ``QuantumBackedButler.retrieve_artifacts`` method to allow dataset artifacts to be retrieved from a graph. (`DM-47328 <https://rubinobs.atlassian.net/browse/DM-47328>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed inserts with ``replace=True`` on dimensions with only primary key columns. (`DM-46631 <https://rubinobs.atlassian.net/browse/DM-46631>`_)
+- Fixed a bug where ``DatastoreCacheManager`` would raise  ``ValueError('badly formed hexadecimal UUID string')`` if files with unexpected names are present in the cache directory when trying to load a file from the cache. (`DM-46936 <https://rubinobs.atlassian.net/browse/DM-46936>`_)
+- Fixed a crash in the new Butler query system which happened in some conditions when using the find-first option with multiple collections. (`DM-47475 <https://rubinobs.atlassian.net/browse/DM-47475>`_)
+- Fixed a bug in which projections spatial-join queries (particularly those where the dimensions of the actual regions being compared are not in the query result rows) could return additional records where there actually was no overlap. (`DM-47947 <https://rubinobs.atlassian.net/browse/DM-47947>`_)
+- Fixed a bug where dataset fields like ``ingest_date`` were raising ``InvalidQueryError: Unrecognized identifier`` when used in a ``Butler.query_datasets`` ``where`` clause. (`DM-48094 <https://rubinobs.atlassian.net/browse/DM-48094>`_)
+- Fixed a query bug that could lead to unexpectedly coarse spatial joins.
+
+  When dataset search or other join operand had some dimensions from either side of a potential spatial join (e.g. ``{tract, visit}``), we had been blocking the addition of an automatic spatial join on the assumption that this would be embedded in that join operand.
+  But this is only desirable when the spatial join that would have been added actually the same one implied by that join operand's dimensions; if it's something more fine grained (e.g. ``{tract, patch, visit}``) we end up with result rows that relate dimensions (e.g. ``patch`` and ``visit``) that do not actually overlap.
+  Now automatic spatial joins are only blocked when the join operand includes all dimensions that would have participated in the automatic join. (`DM-48880 <https://rubinobs.atlassian.net/browse/DM-48880>`_)
+- Fixed a bug that could result in incorrectly empty query results when a data ID constraint was inconsistent with some dataset types in a collection, but not the on actually being queried for. (`DM-48974 <https://rubinobs.atlassian.net/browse/DM-48974>`_)
+- Added pyarrow metadata keywords for astropy table to fix warnings on read. (`DM-49509 <https://rubinobs.atlassian.net/browse/DM-49509>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- Now support a type conversion field in file template format strings. (`DM-47976 <https://rubinobs.atlassian.net/browse/DM-47976>`_)
+- Modified ObsCore configuration to support ``facility_map`` lookup table to allow the facility to be associated with a specific instrument.
+  This is important for butler repositories containing data from multiple instruments and facilities. (`DM-46914 <https://rubinobs.atlassian.net/browse/DM-46914>`_)
+
+
+Butler v28.0.0 (2024-11-20)
+===========================
 
 New Features
 ------------
@@ -135,8 +203,8 @@ An API Removal or Deprecation
   Using ``Butler.collections`` to get the list of default collections is now deprecated.  Use ``Butler.collections.defaults`` instead. (`DM-46599 <https://rubinobs.atlassian.net/browse/DM-46599>`_)
 
 
-Butler 27.0.0 2024-05-28
-========================
+Butler 27.0.0 (2024-05-28)
+==========================
 
 Now supports Python 3.12.
 
@@ -296,8 +364,8 @@ An API Removal or Deprecation
   * Removed the ``reconsitutedDimension`` parameter from ``Quantum.from_simple``. (`DM-40150 <https://rubinobs.atlassian.net/browse/DM-40150>`_)
 
 
-Butler v26.0.0 2023-09-22
-=========================
+Butler v26.0.0 (2023-09-22)
+===========================
 
 Now supports Python 3.11.
 
@@ -475,8 +543,8 @@ An API Removal or Deprecation
 - ``lsst.daf.butler.registry.DbAuth`` class has been moved to the ``lsst-utils`` package and can be imported from the ``lsst.utils.db_auth`` module. (`DM-40462 <https://rubinobs.atlassian.net/browse/DM-40462>`_)
 
 
-Butler v25.0.0 2023-02-27
-=========================
+Butler v25.0.0 (2023-02-27)
+===========================
 
 This is the last release that can access data repositories using integer dataset IDs.
 Please either recreate these repositories or convert them to use UUIDs using `the butler migrate tooling <https://github.com/lsst-dm/daf_butler_migrate>`_.
@@ -570,8 +638,8 @@ An API Removal or Deprecation
 - Removed deprecated filterLabel exposure component access. (`DM-27811 <https://rubinobs.atlassian.net/browse/DM-27811>`_)
 
 
-Butler v24.0.0 2022-08-26
-=========================
+Butler v24.0.0 (2022-08-26)
+===========================
 
 New Features
 ------------
@@ -695,8 +763,8 @@ An API Removal or Deprecation
   It functions identically to the ``filterLabel`` component, which has been deprecated. (`DM-27177 <https://rubinobs.atlassian.net/browse/DM-27177>`_)
 
 
-Butler v23.0.0 2021-12-10
-=========================
+Butler v23.0.0 (2021-12-10)
+===========================
 
 New Features
 ------------
@@ -797,8 +865,8 @@ Other Changes and Additions
 - Add ``split`` transfer mode that can be used when some files are inside the datastore and some files are outside the datastore.
   This is equivalent to using `None` and ``direct`` mode dynamically. (`DM-31251 <https://rubinobs.atlassian.net/browse/DM-31251>`_)
 
-Butler v22.0 2021-04-01
-=======================
+Butler v22.0 (2021-04-01)
+=========================
 
 New Features
 ------------

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/pyproject.toml

@@ -91,7 +91,7 @@ butler = "lsst.daf.butler.cli.butler:main"
     package_dir = "python"
     filename = "doc/lsst.daf.butler/CHANGES.rst"
     directory = "doc/changes"
-    title_format = "Butler {version} {project_date}"
+    title_format = "Butler {version} ({project_date})"
     issue_format = "`{issue} <https://rubinobs.atlassian.net/browse/{issue}>`_"
 
     [[tool.towncrier.type]]

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/arrow_utils.py

@@ -383,7 +383,7 @@ class _ToArrowTimespan(ToArrow):
         # Docstring inherited.
         return TimespanArrowType()
 
-    def append(self, value: Timespan | None, column: list[pa.StructScalar | None]) -> None:
+    def append(self, value: Timespan | None, column: list[dict[str, int] | None]) -> None:
         # Docstring inherited.
         column.append({"begin_nsec": value.nsec[0], "end_nsec": value.nsec[1]} if value is not None else None)
 
@@ -432,12 +432,10 @@ class _ToArrowDateTime(ToArrow):
 
 @final
 class UUIDArrowType(pa.ExtensionType):
-    """An Arrow extension type for `astropy.time.Time`, stored as TAI
-    nanoseconds since 1970-01-01.
-    """
+    """An Arrow extension type for `uuid.UUID`, stored as 16 bytes."""
 
     def __init__(self) -> None:
-        super().__init__(_ToArrowTimespan.storage_type, "astropy.time.Time")
+        super().__init__(_ToArrowUUID.storage_type, "uuid.UUID")
 
     def __arrow_ext_serialize__(self) -> bytes:
         return b""
@@ -458,7 +456,7 @@ class UUIDArrowScalar(pa.ExtensionScalar):
     instance.
     """
 
-    def as_py(self) -> astropy.time.Time:
+    def as_py(self) -> uuid.UUID:
         return uuid.UUID(bytes=self.value.as_py())
 
 

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/column_spec.py

@@ -39,12 +39,24 @@ __all__ = (
     "StringColumnSpec",
     "TimespanColumnSpec",
     "UUIDColumnSpec",
+    "make_tuple_type_adapter",
 )
 
 import textwrap
 import uuid
 from abc import ABC, abstractmethod
-from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Literal, TypeAlias, Union, final
+from collections.abc import Iterable
+from typing import (
+    TYPE_CHECKING,
+    Annotated,
+    Any,
+    ClassVar,
+    Literal,
+    Optional,
+    TypeAlias,
+    Union,
+    final,
+)
 
 import astropy.time
 import pyarrow as pa
@@ -54,7 +66,7 @@ from lsst.sphgeom import Region
 
 from . import arrow_utils, ddl
 from ._timespan import Timespan
-from .pydantic_utils import SerializableRegion, SerializableTime
+from .pydantic_utils import SerializableBytesHex, SerializableRegion, SerializableTime
 
 if TYPE_CHECKING:
     from .name_shrinker import NameShrinker
@@ -125,18 +137,6 @@ class ColumnValueSerializer(ABC):
         raise NotImplementedError
 
 
-class _DefaultColumnValueSerializer(ColumnValueSerializer):
-    """Default implementation of serializer for basic types."""
-
-    def serialize(self, value: Any) -> Any:
-        # Docstring inherited.
-        return value
-
-    def deserialize(self, value: Any) -> Any:
-        # Docstring inherited.
-        return value
-
-
 class _TypeAdapterColumnValueSerializer(ColumnValueSerializer):
     """Implementation of serializer that uses pydantic type adapter."""
 
@@ -156,6 +156,8 @@ class _TypeAdapterColumnValueSerializer(ColumnValueSerializer):
 class _BaseColumnSpec(pydantic.BaseModel, ABC):
     """Base class for descriptions of table columns."""
 
+    pytype: ClassVar[type]
+
     name: str = pydantic.Field(description="""Name of the column.""")
 
     doc: str = pydantic.Field(default="", description="Documentation for the column.")
@@ -200,7 +202,6 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
         """
         raise NotImplementedError()
 
-    @abstractmethod
     def serializer(self) -> ColumnValueSerializer:
         """Return object that converts values of this column to or from
         serializable format.
@@ -210,7 +211,7 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
         serializer : `ColumnValueSerializer`
             A converter instance.
         """
-        raise NotImplementedError()
+        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.annotated_type))
 
     def display(self, level: int = 0, tab: str = "  ") -> list[str]:
         """Return a human-reader-focused string description of this column as
@@ -243,6 +244,48 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
     def __str__(self) -> str:
         return "\n".join(self.display())
 
+    @property
+    def annotated_type(self) -> Any:
+        """Return a Pydantic-friendly type annotation for this column type.
+
+        Since this is a runtime object and most type annotations must be
+        static, this is really only useful for `pydantic.TypeAdapter`
+        construction and dynamic `pydantic.create_model` construction.
+        """
+        base = self._get_base_annotated_type()
+        if self.nullable:
+            return Optional[base]
+        return base
+
+    @abstractmethod
+    def _get_base_annotated_type(self) -> Any:
+        """Return the base annotated type (not taking into account `nullable`)
+        for this column type.
+        """
+        raise NotImplementedError()
+
+
+def make_tuple_type_adapter(
+    columns: Iterable[ColumnSpec],
+) -> pydantic.TypeAdapter[tuple[Any, ...]]:
+    """Return a `pydantic.TypeAdapter` for a `tuple` with types defined by an
+    iterable of `ColumnSpec` objects.
+
+    Parameters
+    ----------
+    columns : `~collections.abc.Iterable` [ `ColumnSpec` ]
+        Iterable of column specifications.
+
+    Returns
+    -------
+    adapter : `pydantic.TypeAdapter`
+        A Pydantic type adapter for the `tuple` representation of a row with
+        the given columns.
+    """
+    # Static type-checkers don't like this runtime use of static-typing
+    # constructs, but that's how Pydantic works.
+    return pydantic.TypeAdapter(tuple[*[spec.annotated_type for spec in columns]])  # type: ignore
+
 
 @final
 class IntColumnSpec(_BaseColumnSpec):
@@ -256,9 +299,9 @@ class IntColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.uint64(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictInt
 
 
 @final
@@ -280,9 +323,9 @@ class StringColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.string(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictStr
 
 
 @final
@@ -310,9 +353,9 @@ class HashColumnSpec(_BaseColumnSpec):
             nullable=self.nullable,
         )
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return SerializableBytesHex
 
 
 @final
@@ -328,9 +371,9 @@ class FloatColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_primitive(self.name, pa.float64(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictFloat
 
 
 @final
@@ -345,9 +388,9 @@ class BoolColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.bool_(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictBool
 
 
 @final
@@ -363,9 +406,9 @@ class UUIDColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_uuid(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.pytype))
+        return uuid.UUID
 
 
 @final
@@ -386,9 +429,9 @@ class RegionColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_region(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(SerializableRegion))
+        return SerializableRegion
 
 
 @final
@@ -405,9 +448,9 @@ class TimespanColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_timespan(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.pytype))
+        return Timespan
 
 
 @final
@@ -425,9 +468,9 @@ class DateTimeColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_datetime(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(SerializableTime))
+        return SerializableTime
 
 
 ColumnSpec = Annotated[

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/dimensions/_coordinate.py

@@ -44,7 +44,7 @@ __all__ = (
 import numbers
 from abc import abstractmethod
 from collections.abc import Iterable, Iterator, Mapping
-from typing import TYPE_CHECKING, Any, ClassVar, overload
+from typing import TYPE_CHECKING, Any, ClassVar, TypeAlias, overload
 
 import pydantic
 
@@ -55,24 +55,13 @@ from .._timespan import Timespan
 from ..json import from_json_pydantic, to_json_pydantic
 from ..persistence_context import PersistenceContextVars
 from ._group import DimensionGroup
-from ._records import DimensionRecord, SerializedDimensionRecord
+from ._records import DataIdKey, DataIdValue, DimensionRecord, SerializedDimensionRecord
 
 if TYPE_CHECKING:  # Imports needed only for type annotations; may be circular.
     from ..registry import Registry
     from ._universe import DimensionUniverse
 
-DataIdKey = str
-"""Type annotation alias for the keys that can be used to index a
-DataCoordinate.
-"""
-
-# Pydantic will cast int to str if str is first in the Union.
-DataIdValue = int | str | None
-"""Type annotation alias for the values that can be present in a
-DataCoordinate or other data ID.
-"""
-
-SerializedDataId = dict[str, DataIdValue]
+SerializedDataId: TypeAlias = dict[str, DataIdValue]
 """Simplified model for serializing the ``mapping`` property of
 `DataCoordinate`.
 """
@@ -247,7 +236,7 @@ class DataCoordinate:
         # Some backends cannot handle numpy.int64 type which is a subclass of
         # numbers.Integral; convert that to int.
         for k, v in new_mapping.items():
-            if isinstance(v, numbers.Integral):
+            if isinstance(v, numbers.Integral):  # type: ignore
                 new_mapping[k] = int(v)  # type: ignore
         if defaults is not None:
             for k, v in defaults.mapping.items():

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/dimensions/_record_set.py

@@ -27,19 +27,22 @@
 
 from __future__ import annotations
 
-__all__ = ("DimensionRecordFactory", "DimensionRecordSet")
+__all__ = ("DimensionRecordFactory", "DimensionRecordSet", "DimensionRecordSetDeserializer")
 
 from collections.abc import Collection, Iterable, Iterator
-from typing import TYPE_CHECKING, Any, Protocol, final
+from typing import TYPE_CHECKING, Any, Protocol, Self, TypeAlias, final
 
 from ._coordinate import DataCoordinate, DataIdValue
-from ._records import DimensionRecord
+from ._records import DimensionRecord, SerializedKeyValueDimensionRecord
 
 if TYPE_CHECKING:
     from ._elements import DimensionElement
     from ._universe import DimensionUniverse
 
 
+SerializedDimensionRecordSetMapping: TypeAlias = dict[str, list[SerializedKeyValueDimensionRecord]]
+
+
 class DimensionRecordFactory(Protocol):
     """Protocol for a callback that can be used to create a dimension record
     to add to a `DimensionRecordSet` when a search for an existing one fails.
@@ -483,3 +486,113 @@ class DimensionRecordSet(Collection[DimensionRecord]):  # numpydoc ignore=PR01
 
     def __deepcopy__(self, memo: dict[str, Any]) -> DimensionRecordSet:
         return DimensionRecordSet(self.element, _by_required_values=self._by_required_values.copy())
+
+    def serialize_records(self) -> list[SerializedKeyValueDimensionRecord]:
+        """Serialize the records to a list.
+
+        Returns
+        -------
+        raw_records : `list` [ `list` ]
+            Serialized records, in the form returned by
+            `DimensionRecord.serialize_key_value`.
+
+        Notes
+        -----
+        This does not include the dimension element shared by all of the
+        records, on the assumption that this is usually more conveniently saved
+        separately (e.g. as the key of a dictionary of which the list of
+        records is a value).
+        """
+        return [record.serialize_key_value() for record in self]
+
+    def deserialize_records(self, raw_records: Iterable[SerializedKeyValueDimensionRecord]) -> None:
+        """Deserialize records and add them to this set.
+
+        Parameters
+        ----------
+        raw_records : `~collections.abc.Iterable` [ `list` ]
+            Serialized records, as returned by `serialize_records` or repeated
+            calls to `DimensionRecord.serialize_key_value`.
+
+        Notes
+        -----
+        The caller is responsible for ensuring that the serialized records have
+        the same dimension element as this set, as this cannot be checked.
+        Mismatches will probably result in a (confusing) type-validation error,
+        but are not guaranteed to.
+        """
+        deserializer = DimensionRecordSetDeserializer.from_raw(self.element, raw_records)
+        self.update(deserializer)
+
+
+class DimensionRecordSetDeserializer:
+    """A helper class for deserializing sets of dimension records, with support
+    for only fully deserializing certain records.
+
+    The `from_raw` factory method should generally be used instead of calling
+    the constructor directly.
+
+    Parameters
+    ----------
+    element : `DimensionElement`
+        Dimension element that defines all records.
+    mapping : `dict` [ `tuple`, `list` ]
+        A dictionary that maps the data ID required-values `tuple` for reach
+        record to the remainder of its raw serialization (i.e. an item in this
+        `dict` is a pair returned by `DimensionRecord.deserialize_key`).  This
+        `dict` will be used directly to back the deserializer, not copied.
+
+    Notes
+    -----
+    The keys (data ID required-values tuples) of all rows are deserialized
+    immediately, but the remaining fields are deserialized only on demand; use
+    `__iter__` to deserialize all records or `__getitem__` to deserialize only
+    a few.  An instance should really only be used for a single iteration or
+    multiple `__getitem__` calls, as each call will re-deserialize the records
+    in play; deserialized records are not cached.
+
+    The caller is responsible for ensuring that the serialized records are for
+    the given dimension element, as this cannot be checked. Mismatches will
+    probably result in a (confusing) type-validation error, but are not
+    guaranteed to.
+    """
+
+    def __init__(
+        self,
+        element: DimensionElement,
+        mapping: dict[tuple[DataIdValue, ...], SerializedKeyValueDimensionRecord],
+    ):
+        self.element = element
+        self._mapping = mapping
+
+    @classmethod
+    def from_raw(
+        cls, element: DimensionElement, raw_records: Iterable[SerializedKeyValueDimensionRecord]
+    ) -> Self:
+        """Construct from raw serialized records.
+
+        Parameters
+        ----------
+        element : `DimensionElement`
+            Dimension element that defines all records.
+        raw_records : `~collections.abc.Iterable` [ `list` ]
+            Serialized records, as returned by
+            `DimensionRecordSet.serialize_records` or repeated calls to
+            `DimensionRecord.serialize_key_value`.
+
+        Returns
+        -------
+        deserializer : `DimensionRecordSetDeserializer`
+            New deserializer instance.
+        """
+        return cls(element=element, mapping=dict(map(element.RecordClass.deserialize_key, raw_records)))
+
+    def __len__(self) -> int:
+        return len(self._mapping)
+
+    def __iter__(self) -> Iterator[DimensionRecord]:
+        deserialize = self.element.RecordClass.deserialize_value
+        return (deserialize(k, v) for k, v in self._mapping.items())
+
+    def __getitem__(self, key: tuple[DataIdValue, ...]) -> DimensionRecord:
+        return self.element.RecordClass.deserialize_value(key, self._mapping[key])

{lsst_daf_butler-29.2025.1200 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/dimensions/_record_table.py

@@ -222,7 +222,7 @@ class DimensionRecordTable:
         array_columns = [
             converter.finish(column) for converter, column in zip(self._converters, list_columns)
         ]
-        return pa.record_batch(array_columns, arrow_schema)
+        return pa.record_batch(array_columns, schema=arrow_schema)
 
     def _get_record_at(self, table: pa.Table | pa.RecordBatch, index: int) -> DimensionRecord:
         """Construct a `DimensionRecord` from a row in the table.