lsst-daf-butler 29.2025.1100__tar.gz → 29.2025.1300__tar.gz

{lsst_daf_butler-29.2025.1100/python/lsst_daf_butler.egg-info → lsst_daf_butler-29.2025.1300}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: lsst-daf-butler
-Version: 29.2025.1100
+Version: 29.2025.1300
 Summary: An abstraction layer for reading and writing astronomical data to datastores.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License: BSD 3-Clause License
@@ -51,6 +51,7 @@ Provides-Extra: https
 Requires-Dist: lsst-resources[https]; extra == "https"
 Provides-Extra: gs
 Requires-Dist: lsst-resources[gs]; extra == "gs"
+Dynamic: license-file
 
 # lsst-daf-butler
 

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/doc/lsst.daf.butler/CHANGES.rst

@@ -1,5 +1,73 @@
-Butler v28.0.0 2024-11-20
-=========================
+Butler v29.0.0 (2025-03-25)
+===========================
+
+New Features
+------------
+
+- * Added new class ``DatasetProvenance`` for tracking the provenance of an individual dataset.
+  * Modified ``Butler.put()`` to accept an optional ``DatasetProvenance``.
+  * Added ``add_provenance`` methods to ``FormatterV2`` and ``StorageClassDelegate``.
+    These methods will now be called with the provenance object during ``Butler.put()`` to allow the in-memory dataset to be updated prior to writing. (`DM-35396 <https://rubinobs.atlassian.net/browse/DM-35396>`_)
+- * Added ``Butler.retrieve_artifacts_zip`` and ``QuantumBackedButler.retrieve_artifacts_zip`` methods to retrieve the dataset artifacts and store them into a zip file.
+  * Added ``Butler.ingest_zip`` to ingest the contents of a Zip file.
+  * Added ``SerializedDatasetRefContainerV1`` class to allow a collection of ``DatasetRef`` to be serialized efficiently.
+    JSON serializations made using this class will be supported.
+  * Added ``--zip`` parameter to ``butler retrieve-artifacts``.
+  * Changed ``Butler.retrieveArtifacts`` to always write a JSON index file describing where the artifacts came from.
+  * Added a ``butler ingest-zip`` command-line tool for ingesting zip files created by ``butler retrieve-artifacts``. (`DM-46776 <https://rubinobs.atlassian.net/browse/DM-46776>`_)
+- The ``DAF_BUTLER_PLUGINS`` environment variable should no longer be set if packages use ``pip install`` and have been upgraded to use entry points.
+  Butler can now read the subcommands from ``pipe_base`` and ``daf_butler_migrate`` automatically.
+  Setting the environment variable for these packages will result in an error. (`DM-47143 <https://rubinobs.atlassian.net/browse/DM-47143>`_)
+- Added two new APIs for handling Butler dataset URIs.
+  ``Butler.parse_dataset_uri`` parses a URI and returns the butler repository label and associated UUID.
+  ``Butler.get_dataset_from_uri`` will parse a URI and attempt to retrieve the ``DatasetRef``.
+  URIs should be in the form of IVOA identifiers as described in `DMTN-302 <https://dmtn-302.lsst.io>`_.
+  Deprecated ``butler://`` URIs are still supported but should not be used in new systems. (`DM-47325 <https://rubinobs.atlassian.net/browse/DM-47325>`_)
+- Added a ``--chains NO-CHILDREN`` mode to the ``butler query-collections`` CLI,
+  which returns results without recursing into ``CHAINED`` collections. (`DM-47768 <https://rubinobs.atlassian.net/browse/DM-47768>`_)
+- Added ``lsst.daf.butler.formatters.parquet.add_pandas_index_to_astropy()`` function which stores special metadata that will be used to create a pandas DataFrame index if the table is read as a ``DataFrame``. (`DM-48141 <https://rubinobs.atlassian.net/browse/DM-48141>`_)
+- Modified the Obscore ``RecordFactory`` to support per-universe subclass discovery using entry points.
+
+  * Added ``RecordFactory.get_record_type_from_universe`` to obtain the correct factory class.
+  * Renamed ``ExposureRegionFactory`` to ``DerivedRegionFactory`` to make it clearer that this class is not solely used for exposures but the usage can change with universe.
+  * Added ``RecordFactory.region_dimension`` to return the dimension that would be needed to obtain a region for this universe. (`DM-48282 <https://rubinobs.atlassian.net/browse/DM-48282>`_)
+- * Added new methods to ``DatasetProvenance`` for serializing provenance to a flat dictionary and recovering provenance from that dictionary.
+  * Modifed ``ParquetFormatter`` to write provenance metadata to Astropy tables. (`DM-48869 <https://rubinobs.atlassian.net/browse/DM-48869>`_)
+
+
+API Changes
+-----------
+
+- Added ``QuantumBackedButler.retrieve_artifacts`` method to allow dataset artifacts to be retrieved from a graph. (`DM-47328 <https://rubinobs.atlassian.net/browse/DM-47328>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed inserts with ``replace=True`` on dimensions with only primary key columns. (`DM-46631 <https://rubinobs.atlassian.net/browse/DM-46631>`_)
+- Fixed a bug where ``DatastoreCacheManager`` would raise  ``ValueError('badly formed hexadecimal UUID string')`` if files with unexpected names are present in the cache directory when trying to load a file from the cache. (`DM-46936 <https://rubinobs.atlassian.net/browse/DM-46936>`_)
+- Fixed a crash in the new Butler query system which happened in some conditions when using the find-first option with multiple collections. (`DM-47475 <https://rubinobs.atlassian.net/browse/DM-47475>`_)
+- Fixed a bug in which projections spatial-join queries (particularly those where the dimensions of the actual regions being compared are not in the query result rows) could return additional records where there actually was no overlap. (`DM-47947 <https://rubinobs.atlassian.net/browse/DM-47947>`_)
+- Fixed a bug where dataset fields like ``ingest_date`` were raising ``InvalidQueryError: Unrecognized identifier`` when used in a ``Butler.query_datasets`` ``where`` clause. (`DM-48094 <https://rubinobs.atlassian.net/browse/DM-48094>`_)
+- Fixed a query bug that could lead to unexpectedly coarse spatial joins.
+
+  When dataset search or other join operand had some dimensions from either side of a potential spatial join (e.g. ``{tract, visit}``), we had been blocking the addition of an automatic spatial join on the assumption that this would be embedded in that join operand.
+  But this is only desirable when the spatial join that would have been added actually the same one implied by that join operand's dimensions; if it's something more fine grained (e.g. ``{tract, patch, visit}``) we end up with result rows that relate dimensions (e.g. ``patch`` and ``visit``) that do not actually overlap.
+  Now automatic spatial joins are only blocked when the join operand includes all dimensions that would have participated in the automatic join. (`DM-48880 <https://rubinobs.atlassian.net/browse/DM-48880>`_)
+- Fixed a bug that could result in incorrectly empty query results when a data ID constraint was inconsistent with some dataset types in a collection, but not the on actually being queried for. (`DM-48974 <https://rubinobs.atlassian.net/browse/DM-48974>`_)
+- Added pyarrow metadata keywords for astropy table to fix warnings on read. (`DM-49509 <https://rubinobs.atlassian.net/browse/DM-49509>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- Now support a type conversion field in file template format strings. (`DM-47976 <https://rubinobs.atlassian.net/browse/DM-47976>`_)
+- Modified ObsCore configuration to support ``facility_map`` lookup table to allow the facility to be associated with a specific instrument.
+  This is important for butler repositories containing data from multiple instruments and facilities. (`DM-46914 <https://rubinobs.atlassian.net/browse/DM-46914>`_)
+
+
+Butler v28.0.0 (2024-11-20)
+===========================
 
 New Features
 ------------
@@ -135,8 +203,8 @@ An API Removal or Deprecation
   Using ``Butler.collections`` to get the list of default collections is now deprecated.  Use ``Butler.collections.defaults`` instead. (`DM-46599 <https://rubinobs.atlassian.net/browse/DM-46599>`_)
 
 
-Butler 27.0.0 2024-05-28
-========================
+Butler 27.0.0 (2024-05-28)
+==========================
 
 Now supports Python 3.12.
 
@@ -296,8 +364,8 @@ An API Removal or Deprecation
   * Removed the ``reconsitutedDimension`` parameter from ``Quantum.from_simple``. (`DM-40150 <https://rubinobs.atlassian.net/browse/DM-40150>`_)
 
 
-Butler v26.0.0 2023-09-22
-=========================
+Butler v26.0.0 (2023-09-22)
+===========================
 
 Now supports Python 3.11.
 
@@ -475,8 +543,8 @@ An API Removal or Deprecation
 - ``lsst.daf.butler.registry.DbAuth`` class has been moved to the ``lsst-utils`` package and can be imported from the ``lsst.utils.db_auth`` module. (`DM-40462 <https://rubinobs.atlassian.net/browse/DM-40462>`_)
 
 
-Butler v25.0.0 2023-02-27
-=========================
+Butler v25.0.0 (2023-02-27)
+===========================
 
 This is the last release that can access data repositories using integer dataset IDs.
 Please either recreate these repositories or convert them to use UUIDs using `the butler migrate tooling <https://github.com/lsst-dm/daf_butler_migrate>`_.
@@ -570,8 +638,8 @@ An API Removal or Deprecation
 - Removed deprecated filterLabel exposure component access. (`DM-27811 <https://rubinobs.atlassian.net/browse/DM-27811>`_)
 
 
-Butler v24.0.0 2022-08-26
-=========================
+Butler v24.0.0 (2022-08-26)
+===========================
 
 New Features
 ------------
@@ -695,8 +763,8 @@ An API Removal or Deprecation
   It functions identically to the ``filterLabel`` component, which has been deprecated. (`DM-27177 <https://rubinobs.atlassian.net/browse/DM-27177>`_)
 
 
-Butler v23.0.0 2021-12-10
-=========================
+Butler v23.0.0 (2021-12-10)
+===========================
 
 New Features
 ------------
@@ -797,8 +865,8 @@ Other Changes and Additions
 - Add ``split`` transfer mode that can be used when some files are inside the datastore and some files are outside the datastore.
   This is equivalent to using `None` and ``direct`` mode dynamically. (`DM-31251 <https://rubinobs.atlassian.net/browse/DM-31251>`_)
 
-Butler v22.0 2021-04-01
-=======================
+Butler v22.0 (2021-04-01)
+=========================
 
 New Features
 ------------

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/pyproject.toml

@@ -91,7 +91,7 @@ butler = "lsst.daf.butler.cli.butler:main"
     package_dir = "python"
     filename = "doc/lsst.daf.butler/CHANGES.rst"
     directory = "doc/changes"
-    title_format = "Butler {version} {project_date}"
+    title_format = "Butler {version} ({project_date})"
     issue_format = "`{issue} <https://rubinobs.atlassian.net/browse/{issue}>`_"
 
     [[tool.towncrier.type]]

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/arrow_utils.py

@@ -383,7 +383,7 @@ class _ToArrowTimespan(ToArrow):
         # Docstring inherited.
         return TimespanArrowType()
 
-    def append(self, value: Timespan | None, column: list[pa.StructScalar | None]) -> None:
+    def append(self, value: Timespan | None, column: list[dict[str, int] | None]) -> None:
         # Docstring inherited.
         column.append({"begin_nsec": value.nsec[0], "end_nsec": value.nsec[1]} if value is not None else None)
 
@@ -432,12 +432,10 @@ class _ToArrowDateTime(ToArrow):
 
 @final
 class UUIDArrowType(pa.ExtensionType):
-    """An Arrow extension type for `astropy.time.Time`, stored as TAI
-    nanoseconds since 1970-01-01.
-    """
+    """An Arrow extension type for `uuid.UUID`, stored as 16 bytes."""
 
     def __init__(self) -> None:
-        super().__init__(_ToArrowTimespan.storage_type, "astropy.time.Time")
+        super().__init__(_ToArrowUUID.storage_type, "uuid.UUID")
 
     def __arrow_ext_serialize__(self) -> bytes:
         return b""
@@ -458,7 +456,7 @@ class UUIDArrowScalar(pa.ExtensionScalar):
     instance.
     """
 
-    def as_py(self) -> astropy.time.Time:
+    def as_py(self) -> uuid.UUID:
         return uuid.UUID(bytes=self.value.as_py())
 
 

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/cli/cmd/commands.py

@@ -638,6 +638,9 @@ def retrieve_artifacts(**kwargs: Any) -> None:
 @transfer_option()
 @register_dataset_types_option()
 @transfer_dimensions_option()
+@click.option(
+    "--dry-run/--no-dry-run", default=False, help="Enable dry run mode and do not transfer any datasets."
+)
 @options_file_option()
 def transfer_datasets(**kwargs: Any) -> None:
     """Transfer datasets from a source butler to a destination butler.

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/column_spec.py

@@ -39,12 +39,24 @@ __all__ = (
     "StringColumnSpec",
     "TimespanColumnSpec",
     "UUIDColumnSpec",
+    "make_tuple_type_adapter",
 )
 
 import textwrap
 import uuid
 from abc import ABC, abstractmethod
-from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Literal, TypeAlias, Union, final
+from collections.abc import Iterable
+from typing import (
+    TYPE_CHECKING,
+    Annotated,
+    Any,
+    ClassVar,
+    Literal,
+    Optional,
+    TypeAlias,
+    Union,
+    final,
+)
 
 import astropy.time
 import pyarrow as pa
@@ -54,7 +66,7 @@ from lsst.sphgeom import Region
 
 from . import arrow_utils, ddl
 from ._timespan import Timespan
-from .pydantic_utils import SerializableRegion, SerializableTime
+from .pydantic_utils import SerializableBytesHex, SerializableRegion, SerializableTime
 
 if TYPE_CHECKING:
     from .name_shrinker import NameShrinker
@@ -125,18 +137,6 @@ class ColumnValueSerializer(ABC):
         raise NotImplementedError
 
 
-class _DefaultColumnValueSerializer(ColumnValueSerializer):
-    """Default implementation of serializer for basic types."""
-
-    def serialize(self, value: Any) -> Any:
-        # Docstring inherited.
-        return value
-
-    def deserialize(self, value: Any) -> Any:
-        # Docstring inherited.
-        return value
-
-
 class _TypeAdapterColumnValueSerializer(ColumnValueSerializer):
     """Implementation of serializer that uses pydantic type adapter."""
 
@@ -156,6 +156,8 @@ class _TypeAdapterColumnValueSerializer(ColumnValueSerializer):
 class _BaseColumnSpec(pydantic.BaseModel, ABC):
     """Base class for descriptions of table columns."""
 
+    pytype: ClassVar[type]
+
     name: str = pydantic.Field(description="""Name of the column.""")
 
     doc: str = pydantic.Field(default="", description="Documentation for the column.")
@@ -200,7 +202,6 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
         """
         raise NotImplementedError()
 
-    @abstractmethod
     def serializer(self) -> ColumnValueSerializer:
         """Return object that converts values of this column to or from
         serializable format.
@@ -210,7 +211,7 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
         serializer : `ColumnValueSerializer`
             A converter instance.
         """
-        raise NotImplementedError()
+        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.annotated_type))
 
     def display(self, level: int = 0, tab: str = "  ") -> list[str]:
         """Return a human-reader-focused string description of this column as
@@ -243,6 +244,48 @@ class _BaseColumnSpec(pydantic.BaseModel, ABC):
     def __str__(self) -> str:
         return "\n".join(self.display())
 
+    @property
+    def annotated_type(self) -> Any:
+        """Return a Pydantic-friendly type annotation for this column type.
+
+        Since this is a runtime object and most type annotations must be
+        static, this is really only useful for `pydantic.TypeAdapter`
+        construction and dynamic `pydantic.create_model` construction.
+        """
+        base = self._get_base_annotated_type()
+        if self.nullable:
+            return Optional[base]
+        return base
+
+    @abstractmethod
+    def _get_base_annotated_type(self) -> Any:
+        """Return the base annotated type (not taking into account `nullable`)
+        for this column type.
+        """
+        raise NotImplementedError()
+
+
+def make_tuple_type_adapter(
+    columns: Iterable[ColumnSpec],
+) -> pydantic.TypeAdapter[tuple[Any, ...]]:
+    """Return a `pydantic.TypeAdapter` for a `tuple` with types defined by an
+    iterable of `ColumnSpec` objects.
+
+    Parameters
+    ----------
+    columns : `~collections.abc.Iterable` [ `ColumnSpec` ]
+        Iterable of column specifications.
+
+    Returns
+    -------
+    adapter : `pydantic.TypeAdapter`
+        A Pydantic type adapter for the `tuple` representation of a row with
+        the given columns.
+    """
+    # Static type-checkers don't like this runtime use of static-typing
+    # constructs, but that's how Pydantic works.
+    return pydantic.TypeAdapter(tuple[*[spec.annotated_type for spec in columns]])  # type: ignore
+
 
 @final
 class IntColumnSpec(_BaseColumnSpec):
@@ -256,9 +299,9 @@ class IntColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.uint64(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictInt
 
 
 @final
@@ -280,9 +323,9 @@ class StringColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.string(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictStr
 
 
 @final
@@ -310,9 +353,9 @@ class HashColumnSpec(_BaseColumnSpec):
             nullable=self.nullable,
         )
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return SerializableBytesHex
 
 
 @final
@@ -328,9 +371,9 @@ class FloatColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_primitive(self.name, pa.float64(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictFloat
 
 
 @final
@@ -345,9 +388,9 @@ class BoolColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_primitive(self.name, pa.bool_(), nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _DefaultColumnValueSerializer()
+        return pydantic.StrictBool
 
 
 @final
@@ -363,9 +406,9 @@ class UUIDColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_uuid(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.pytype))
+        return uuid.UUID
 
 
 @final
@@ -386,9 +429,9 @@ class RegionColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_region(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(SerializableRegion))
+        return SerializableRegion
 
 
 @final
@@ -405,9 +448,9 @@ class TimespanColumnSpec(_BaseColumnSpec):
         # Docstring inherited.
         return arrow_utils.ToArrow.for_timespan(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(self.pytype))
+        return Timespan
 
 
 @final
@@ -425,9 +468,9 @@ class DateTimeColumnSpec(_BaseColumnSpec):
         assert self.nullable is not None, "nullable=None should be resolved by validators"
         return arrow_utils.ToArrow.for_datetime(self.name, nullable=self.nullable)
 
-    def serializer(self) -> ColumnValueSerializer:
+    def _get_base_annotated_type(self) -> Any:
         # Docstring inherited.
-        return _TypeAdapterColumnValueSerializer(pydantic.TypeAdapter(SerializableTime))
+        return SerializableTime
 
 
 ColumnSpec = Annotated[

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/datastore/_datastore.py

@@ -137,6 +137,10 @@ class DatastoreTransaction:
     ----------
     parent : `DatastoreTransaction`, optional
         The parent transaction (if any).
+
+    Notes
+    -----
+    This transaction object must be thread safe.
     """
 
     Event: ClassVar[type] = Event

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/datastores/fileDatastore.py

@@ -2158,10 +2158,15 @@ class FileDatastore(GenericBaseDatastore[StoredFileInfo]):
                     target_uri = to_transfer[cleaned_source_uri]
                     artifact_map[target_uri].append(ref.id)
 
-        # In theory can now parallelize the transfer
+        # Parallelize the transfer. Re-raise as a single exception if
+        # a FileExistsError is encountered anywhere.
         log.debug("Number of artifacts to transfer to %s: %d", str(destination), len(to_transfer))
-        for source_uri, target_uri in to_transfer.items():
-            target_uri.transfer_from(source_uri, transfer=transfer, overwrite=overwrite)
+        try:
+            ResourcePath.mtransfer(transfer, tuple(to_transfer.items()), overwrite=overwrite)
+        except* FileExistsError as egroup:
+            raise FileExistsError(
+                "Some files already exist in destination directory and overwrite is False"
+            ) from egroup
 
         # Transfer the Zip files and unpack them.
         zipped_artifacts = unpack_zips(zips_to_transfer, requested_ids, destination, preserve_path, overwrite)
@@ -2748,6 +2753,9 @@ class FileDatastore(GenericBaseDatastore[StoredFileInfo]):
         # Record each time we have done a "direct" transfer.
         direct_transfers = []
 
+        # Keep track of all the file transfers that are required.
+        from_to: list[tuple[ResourcePath, ResourcePath]] = []
+
         # Now can transfer the artifacts
         log.verbose("Transferring artifacts")
         for ref in refs:
@@ -2796,21 +2804,28 @@ class FileDatastore(GenericBaseDatastore[StoredFileInfo]):
                             info = info.update(path=target_location.pathInStore.path)
 
                     # Need to transfer it to the new location.
-                    # Assume we should always overwrite. If the artifact
-                    # is there this might indicate that a previous transfer
-                    # was interrupted but was not able to be rolled back
-                    # completely (eg pre-emption) so follow Datastore default
-                    # and overwrite. Do not copy if we are in dry-run mode.
-                    if not dry_run:
-                        target_location.uri.transfer_from(
-                            source_location.uri,
-                            transfer=transfer,
-                            overwrite=True,
-                            transaction=self._transaction,
-                        )
+                    from_to.append((source_location.uri, target_location.uri))
 
                 artifacts.append((ref, info))
 
+        # Do the file transfers in bulk.
+        # Assume we should always overwrite. If the artifact
+        # is there this might indicate that a previous transfer
+        # was interrupted but was not able to be rolled back
+        # completely (eg pre-emption) so follow Datastore default
+        # and overwrite. Do not copy if we are in dry-run mode.
+        if dry_run:
+            log.info("Would be copying %d file artifacts", len(from_to))
+        else:
+            log.verbose("Copying %d file artifacts", len(from_to))
+            with time_this(log, msg="Transferring datasets into datastore", level=VERBOSE):
+                ResourcePath.mtransfer(
+                    transfer,
+                    from_to,
+                    overwrite=True,
+                    transaction=self._transaction,
+                )
+
         if direct_transfers:
             log.info(
                 "Transfer request for an outside-datastore artifact with absolute URI done %d time%s",

{lsst_daf_butler-29.2025.1100 → lsst_daf_butler-29.2025.1300}/python/lsst/daf/butler/dimensions/_coordinate.py

@@ -44,7 +44,7 @@ __all__ = (
 import numbers
 from abc import abstractmethod
 from collections.abc import Iterable, Iterator, Mapping
-from typing import TYPE_CHECKING, Any, ClassVar, overload
+from typing import TYPE_CHECKING, Any, ClassVar, TypeAlias, overload
 
 import pydantic
 
@@ -55,24 +55,13 @@ from .._timespan import Timespan
 from ..json import from_json_pydantic, to_json_pydantic
 from ..persistence_context import PersistenceContextVars
 from ._group import DimensionGroup
-from ._records import DimensionRecord, SerializedDimensionRecord
+from ._records import DataIdKey, DataIdValue, DimensionRecord, SerializedDimensionRecord
 
 if TYPE_CHECKING:  # Imports needed only for type annotations; may be circular.
     from ..registry import Registry
     from ._universe import DimensionUniverse
 
-DataIdKey = str
-"""Type annotation alias for the keys that can be used to index a
-DataCoordinate.
-"""
-
-# Pydantic will cast int to str if str is first in the Union.
-DataIdValue = int | str | None
-"""Type annotation alias for the values that can be present in a
-DataCoordinate or other data ID.
-"""
-
-SerializedDataId = dict[str, DataIdValue]
+SerializedDataId: TypeAlias = dict[str, DataIdValue]
 """Simplified model for serializing the ``mapping`` property of
 `DataCoordinate`.
 """
@@ -247,7 +236,7 @@ class DataCoordinate:
         # Some backends cannot handle numpy.int64 type which is a subclass of
         # numbers.Integral; convert that to int.
         for k, v in new_mapping.items():
-            if isinstance(v, numbers.Integral):
+            if isinstance(v, numbers.Integral):  # type: ignore
                 new_mapping[k] = int(v)  # type: ignore
         if defaults is not None:
             for k, v in defaults.mapping.items():