lsst-daf-butler 29.0.1rc1__tar.gz → 29.1.0rc2__tar.gz

{lsst_daf_butler-29.0.1rc1/python/lsst_daf_butler.egg-info → lsst_daf_butler-29.1.0rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lsst-daf-butler
-Version: 29.0.1rc1
+Version: 29.1.0rc2
 Summary: An abstraction layer for reading and writing astronomical data to datastores.
 Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
 License: BSD 3-Clause License

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/doc/lsst.daf.butler/CHANGES.rst

@@ -1,5 +1,5 @@
-Butler v29.0.0 2025-03-25
-=========================
+Butler v29.0.0 (2025-03-25)
+===========================
 
 New Features
 ------------
@@ -66,8 +66,8 @@ Other Changes and Additions
   This is important for butler repositories containing data from multiple instruments and facilities. (`DM-46914 <https://rubinobs.atlassian.net/browse/DM-46914>`_)
 
 
-Butler v28.0.0 2024-11-20
-=========================
+Butler v28.0.0 (2024-11-20)
+===========================
 
 New Features
 ------------
@@ -203,8 +203,8 @@ An API Removal or Deprecation
   Using ``Butler.collections`` to get the list of default collections is now deprecated.  Use ``Butler.collections.defaults`` instead. (`DM-46599 <https://rubinobs.atlassian.net/browse/DM-46599>`_)
 
 
-Butler 27.0.0 2024-05-28
-========================
+Butler 27.0.0 (2024-05-28)
+==========================
 
 Now supports Python 3.12.
 
@@ -364,8 +364,8 @@ An API Removal or Deprecation
   * Removed the ``reconsitutedDimension`` parameter from ``Quantum.from_simple``. (`DM-40150 <https://rubinobs.atlassian.net/browse/DM-40150>`_)
 
 
-Butler v26.0.0 2023-09-22
-=========================
+Butler v26.0.0 (2023-09-22)
+===========================
 
 Now supports Python 3.11.
 
@@ -543,8 +543,8 @@ An API Removal or Deprecation
 - ``lsst.daf.butler.registry.DbAuth`` class has been moved to the ``lsst-utils`` package and can be imported from the ``lsst.utils.db_auth`` module. (`DM-40462 <https://rubinobs.atlassian.net/browse/DM-40462>`_)
 
 
-Butler v25.0.0 2023-02-27
-=========================
+Butler v25.0.0 (2023-02-27)
+===========================
 
 This is the last release that can access data repositories using integer dataset IDs.
 Please either recreate these repositories or convert them to use UUIDs using `the butler migrate tooling <https://github.com/lsst-dm/daf_butler_migrate>`_.
@@ -638,8 +638,8 @@ An API Removal or Deprecation
 - Removed deprecated filterLabel exposure component access. (`DM-27811 <https://rubinobs.atlassian.net/browse/DM-27811>`_)
 
 
-Butler v24.0.0 2022-08-26
-=========================
+Butler v24.0.0 (2022-08-26)
+===========================
 
 New Features
 ------------
@@ -763,8 +763,8 @@ An API Removal or Deprecation
   It functions identically to the ``filterLabel`` component, which has been deprecated. (`DM-27177 <https://rubinobs.atlassian.net/browse/DM-27177>`_)
 
 
-Butler v23.0.0 2021-12-10
-=========================
+Butler v23.0.0 (2021-12-10)
+===========================
 
 New Features
 ------------
@@ -865,8 +865,8 @@ Other Changes and Additions
 - Add ``split`` transfer mode that can be used when some files are inside the datastore and some files are outside the datastore.
   This is equivalent to using `None` and ``direct`` mode dynamically. (`DM-31251 <https://rubinobs.atlassian.net/browse/DM-31251>`_)
 
-Butler v22.0 2021-04-01
-=======================
+Butler v22.0 (2021-04-01)
+=========================
 
 New Features
 ------------

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/doc/lsst.daf.butler/queries.rst

@@ -136,6 +136,16 @@ Time literals
     time scale. For detailed description of supported time specification
     check section :ref:`time-literals-syntax`.
 
+UUID literals
+    UUID values are produced by ``UUID()`` function which accepts a string
+    representation of the UUID. The string can have any format acceptable
+    by Python ``uuid.UUID()`` method.
+
+    Examples:
+
+    - ``UUID('12345678-1234-5678-1234-567812345678')``
+    - ``UUID('12345678123456781234567812345678')``
+
 Range literals
     This sort of literal is allowed inside ``IN`` expressions only. It consists
     of two integer literals separated by double dots and optionally followed by
@@ -167,6 +177,8 @@ filter query results based on that property of datasets.
 
 Registry methods accepting user expressions also accept a ``bind`` parameter, which is a mapping from identifier name to its corresponding value.
 Identifiers appearing in user expressions will be replaced with the corresponding value from this mapping.
+Bind identifiers in expressions are specified with a preceding colon (e.g. ``:id``).
+The legacy format where bind identifiers can be specified without a colon is still supported, but discouraged, and will be deprecated and removed in the future.
 Using the ``bind`` parameter is encouraged when possible to simplify rendering of the query strings.
 A partial example of comparing two approaches, without and with ``bind``:
 
@@ -186,7 +198,7 @@ A partial example of comparing two approaches, without and with ``bind``:
     # Same functionality using bind parameter
     result = butler.query_datasets(
         dataset_type,
-        where="instrument = instrument_name AND visit = visit_id",
+        where="instrument = :instrument_name AND visit = :visit_id",
         bind={"instrument_name": instrument_name, "visit_id": visit_id},
     )
 
@@ -201,7 +213,7 @@ An example of this feature:
     visit_ids = (12345, 12346, 12350)
     result = butler.query_datasets(
         dataset_type,
-        where="instrument = instrument_name AND visit IN (visit_ids)",
+        where="instrument = :instrument_name AND visit IN (:visit_ids)",
         bind={"instrument_name": instrument_name, "visit_ids": visit_ids},
     )
 
@@ -227,7 +239,7 @@ Comparison operators
 
 Language supports set of regular comparison operators: ``=``, ``!=``, ``<``,
 ``<=``, ``>``, ``>=``. This can be used on operands that evaluate to a numeric
-values or timestamps.
+values, timestamps, or strings.
 
 .. note :: The equality comparison operator is a single ``=`` like in SQL, not
     double ``==`` like in Python or C++.
@@ -270,7 +282,7 @@ as are these:
 
 Another usage of ``IN`` operator is for checking whether a timestamp or a time
 range is contained wholly in other time range. Time range in this case can be
-specified as a tuple of two time literals or identifers each representing a
+specified as a tuple of two time literals or identifiers each representing a
 timestamp, or as a single identifier representing a time range. In case a
 single identifier appears on the right side of ``IN`` it has to be enclosed
 in parentheses.
@@ -350,11 +362,17 @@ Function call
 Function call syntax is similar to other languages, expression for call
 consists of an identifier followed by zero or more comma-separated arguments
 enclosed in parentheses (e.g. ``func(1, 2, 3)``). An argument to a function
-can be any expression.
+can be any expression. Names of the functions are not case-sensitive.
+
+Presently the following functions are implemented in the query language:
 
-Presently there only one construct that uses this syntax, ``POINT(ra, dec)``
-is function which declares (or returns) sky coordinates similarly to ADQL
-syntax. Name of the ``POINT`` function is not case-sensitive.
+- ``POINT(ra, dec)`` - function which declares (or returns) sky coordinates
+  similarly to ADQL syntax.
+- ``GLOB(expression, pattern)`` - performs case-sensitive match of a string
+  ``expression`` against ``pattern``. Pattern can include ``*`` and ``?``
+  meta-characters, matching any number of characters or a single character
+  respectfully. All other characters in pattern are matched literally, and
+  whole expression string has to match for ``GLOB()`` to return ``True``.
 
 
 .. _time-literals-syntax:
@@ -434,6 +452,8 @@ Few examples of valid expressions using some of the constructs:
 
     ingest_date < T'2020-11-06 21:10:00'
 
+    flat.dataset_id = UUID('e15ab039-bc8b-4135-87c5-90902a7c0b22')
+
 
 .. _daf_butler_query_ordering:
 

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/pyproject.toml

@@ -91,7 +91,7 @@ butler = "lsst.daf.butler.cli.butler:main"
     package_dir = "python"
     filename = "doc/lsst.daf.butler/CHANGES.rst"
     directory = "doc/changes"
-    title_format = "Butler {version} {project_date}"
+    title_format = "Butler {version} ({project_date})"
     issue_format = "`{issue} <https://rubinobs.atlassian.net/browse/{issue}>`_"
 
     [[tool.towncrier.type]]

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/python/lsst/daf/butler/__init__.py

@@ -38,6 +38,7 @@ from . import ddl, time_utils
 from ._butler import *
 from ._butler_collections import *
 from ._butler_config import *
+from ._butler_metrics import *
 from ._butler_repo_index import *
 from ._collection_type import CollectionType
 from ._column_categorization import *

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/python/lsst/daf/butler/_butler.py

@@ -46,6 +46,7 @@ from lsst.utils.logging import getLogger
 from ._butler_collections import ButlerCollections
 from ._butler_config import ButlerConfig, ButlerType
 from ._butler_instance_options import ButlerInstanceOptions
+from ._butler_metrics import ButlerMetrics
 from ._butler_repo_index import ButlerRepoIndex
 from ._config import Config, ConfigSubset
 from ._exceptions import EmptyQueryResultError, InvalidQueryError
@@ -89,6 +90,10 @@ class SpecificButlerDataset:
     dataset: DatasetRef | None
 
 
+class _DeprecatedDefault:
+    """Default value for a deprecated parameter."""
+
+
 class Butler(LimitedButler):  # numpydoc ignore=PR02
     """Interface for data butler and factory for Butler instances.
 
@@ -154,6 +159,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         writeable: bool | None = None,
         inferDefaults: bool = True,
         without_datastore: bool = False,
+        metrics: ButlerMetrics | None = None,
         **kwargs: Any,
     ) -> Butler:
         if cls is Butler:
@@ -165,6 +171,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
                 writeable=writeable,
                 inferDefaults=inferDefaults,
                 without_datastore=without_datastore,
+                metrics=metrics,
                 **kwargs,
             )
 
@@ -183,6 +190,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         writeable: bool | None = None,
         inferDefaults: bool = True,
         without_datastore: bool = False,
+        metrics: ButlerMetrics | None = None,
         **kwargs: Any,
     ) -> Butler:
         """Create butler instance from configuration.
@@ -230,6 +238,8 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         without_datastore : `bool`, optional
             If `True` do not attach a datastore to this butler. Any attempts
             to use a datastore will fail.
+        metrics : `ButlerMetrics` or `None`, optional
+            Metrics object to record butler usage statistics.
         **kwargs : `Any`
             Default data ID key-value pairs.  These may only identify
             "governor" dimensions like ``instrument`` and ``skymap``.
@@ -300,6 +310,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         # passing the "butler" parameter to its constructor.  This has
         # been moved out of the constructor into Butler.clone().
         butler = kwargs.pop("butler", None)
+        metrics = metrics if metrics is not None else ButlerMetrics()
         if butler is not None:
             if not isinstance(butler, Butler):
                 raise TypeError("'butler' parameter must be a Butler instance")
@@ -307,10 +318,17 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
                 raise TypeError(
                     "Cannot pass 'config', 'searchPaths', or 'writeable' arguments with 'butler' argument."
                 )
-            return butler.clone(collections=collections, run=run, inferDefaults=inferDefaults, dataId=kwargs)
+            return butler.clone(
+                collections=collections, run=run, inferDefaults=inferDefaults, metrics=metrics, dataId=kwargs
+            )
 
         options = ButlerInstanceOptions(
-            collections=collections, run=run, writeable=writeable, inferDefaults=inferDefaults, kwargs=kwargs
+            collections=collections,
+            run=run,
+            writeable=writeable,
+            inferDefaults=inferDefaults,
+            metrics=metrics,
+            kwargs=kwargs,
         )
 
         # Load the Butler configuration.  This may involve searching the
@@ -1274,7 +1292,13 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         raise NotImplementedError()
 
     @abstractmethod
-    def removeRuns(self, names: Iterable[str], unstore: bool = True) -> None:
+    def removeRuns(
+        self,
+        names: Iterable[str],
+        unstore: bool | type[_DeprecatedDefault] = _DeprecatedDefault,
+        *,
+        unlink_from_chains: bool = False,
+    ) -> None:
         """Remove one or more `~CollectionType.RUN` collections and the
         datasets within them.
 
@@ -1287,7 +1311,13 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
             they are present, and attempt to rollback the registry deletions if
             datastore deletions fail (which may not always be possible).  If
             `False`, datastore records for these datasets are still removed,
-            but any artifacts (e.g. files) will not be.
+            but any artifacts (e.g. files) will not be. This parameter is now
+            deprecated and no longer has any effect. Files are always deleted
+            from datastores unless they were ingested using full URIs.
+        unlink_from_chains : `bool`, optional
+            If `True` remove the RUN collection from any chains prior to
+            removing the RUN. If `False` the removal will fail if any chains
+            still refer to the RUN.
 
         Raises
         ------
@@ -1360,7 +1390,14 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         raise NotImplementedError()
 
     @abstractmethod
-    def ingest_zip(self, zip_file: ResourcePathExpression, transfer: str = "auto") -> None:
+    def ingest_zip(
+        self,
+        zip_file: ResourcePathExpression,
+        transfer: str = "auto",
+        *,
+        transfer_dimensions: bool = False,
+        dry_run: bool = False,
+    ) -> None:
         """Ingest a Zip file into this butler.
 
         The Zip file must have been created by `retrieve_artifacts_zip`.
@@ -1371,10 +1408,17 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
             Path to the Zip file.
         transfer : `str`, optional
             Method to use to transfer the Zip into the datastore.
+        transfer_dimensions : `bool`, optional
+            If `True`, dimension record data associated with the new datasets
+            will be transferred from the Zip file, if present.
+        dry_run : `bool`, optional
+            If `True` the ingest will be processed without any modifications
+            made to the target butler and as if the target butler did not
+            have any of the datasets.
 
         Notes
         -----
-        Run collections are created as needed.
+        Run collections and dataset types are created as needed.
         """
         raise NotImplementedError()
 
@@ -1741,8 +1785,8 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
             warn_limit = True
         with self.query() as query:
             result = (
-                query.where(data_id, where, bind=bind, **kwargs)
-                .data_ids(dimensions)
+                query.data_ids(dimensions)
+                .where(data_id, where, bind=bind, **kwargs)
                 .order_by(*ensure_iterable(order_by))
                 .limit(query_limit)
             )
@@ -1979,8 +2023,8 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
             warn_limit = True
         with self.query() as query:
             result = (
-                query.where(data_id, where, bind=bind, **kwargs)
-                .dimension_records(element)
+                query.dimension_records(element)
+                .where(data_id, where, bind=bind, **kwargs)
                 .order_by(*ensure_iterable(order_by))
                 .limit(query_limit)
             )
@@ -2123,6 +2167,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         run: str | None | EllipsisType = ...,
         inferDefaults: bool | EllipsisType = ...,
         dataId: dict[str, str] | EllipsisType = ...,
+        metrics: ButlerMetrics | None = None,
     ) -> Butler:
         """Return a new Butler instance connected to the same repository
         as this one, optionally overriding ``collections``, ``run``,
@@ -2142,5 +2187,7 @@ class Butler(LimitedButler):  # numpydoc ignore=PR02
         dataId : `str`
             Same as ``kwargs`` passed to the constructor.  If omitted, copies
             values from original object.
+        metrics : `ButlerMetrics` or `None`, optional
+            Metrics object to record butler statistics.
         """
         raise NotImplementedError()

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/python/lsst/daf/butler/_butler_collections.py

@@ -108,6 +108,10 @@ class ButlerCollections(ABC, Sequence):
         """Collection defaults associated with this butler."""
         raise NotImplementedError("Defaults must be implemented by a subclass")
 
+    def __str__(self) -> str:
+        """Return string representation."""
+        return f"{self.__class__.__name__}(defaults={self.defaults})"
+
     @abstractmethod
     def extend_chain(self, parent_collection_name: str, child_collection_names: str | Iterable[str]) -> None:
         """Add children to the end of a CHAINED collection.

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/python/lsst/daf/butler/_butler_instance_options.py

@@ -30,6 +30,8 @@ __all__ = ("ButlerInstanceOptions",)
 import dataclasses
 from typing import Any
 
+from ._butler_metrics import ButlerMetrics
+
 
 @dataclasses.dataclass(frozen=True)
 class ButlerInstanceOptions:
@@ -43,4 +45,5 @@ class ButlerInstanceOptions:
     run: str | None = None
     writeable: bool | None = None
     inferDefaults: bool = True
+    metrics: ButlerMetrics = dataclasses.field(default_factory=ButlerMetrics)
     kwargs: dict[str, Any] = dataclasses.field(default_factory=dict)

lsst_daf_butler-29.1.0rc2/python/lsst/daf/butler/_butler_metrics.py

@@ -0,0 +1,117 @@
+# This file is part of daf_butler.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (http://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This software is dual licensed under the GNU General Public License and also
+# under a 3-clause BSD license. Recipients may choose which of these licenses
+# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
+# respectively.  If you choose the GPL option then the following text applies
+# (but note that there is still no warranty even if you opt for BSD instead):
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+
+from pydantic import BaseModel
+
+from lsst.utils.logging import LsstLoggers
+from lsst.utils.timer import time_this
+
+
+class ButlerMetrics(BaseModel):
+    """Metrics collected during Butler operations."""
+
+    time_in_put: float = 0.0
+    """Wall-clock time, in seconds, spent in put()."""
+
+    time_in_get: float = 0.0
+    """Wall-clock time, in seconds, spent in get()."""
+
+    n_get: int = 0
+    """Number of datasets retrieved with get()."""
+
+    n_put: int = 0
+    """Number of datasets stored with put()."""
+
+    def reset(self) -> None:
+        """Reset all metrics."""
+        self.time_in_put = 0.0
+        self.time_in_get = 0.0
+        self.n_get = 0
+        self.n_put = 0
+
+    def increment_get(self, duration: float) -> None:
+        """Increment time for get().
+
+        Parameters
+        ----------
+        duration : `float`
+            Duration to add to the get() statistics.
+        """
+        self.time_in_get += duration
+        self.n_get += 1
+
+    def increment_put(self, duration: float) -> None:
+        """Increment time for put().
+
+        Parameters
+        ----------
+        duration : `float`
+            Duration to add to the put() statistics.
+        """
+        self.time_in_put += duration
+        self.n_put += 1
+
+    @contextmanager
+    def _timer(
+        self, handler: Callable[[float], None], log: LsstLoggers | None = None, msg: str | None = None
+    ) -> Iterator[None]:
+        with time_this(log=log, msg=msg) as timer:
+            yield
+        handler(timer.duration)
+
+    @contextmanager
+    def instrument_get(self, log: LsstLoggers | None = None, msg: str | None = None) -> Iterator[None]:
+        """Run code and increment get statistics.
+
+        Parameters
+        ----------
+        log : `logging.Logger` or `None`
+            Logger to use for any timing information.
+        msg : `str` or `None`
+            Any message to be included in log output.
+        """
+        with self._timer(self.increment_get, log=log, msg=msg):
+            yield
+
+    @contextmanager
+    def instrument_put(self, log: LsstLoggers | None = None, msg: str | None = None) -> Iterator[None]:
+        """Run code and increment put statistics.
+
+        Parameters
+        ----------
+        log : `logging.Logger` or `None`
+            Logger to use for any timing information.
+        msg : `str` or `None`
+            Any message to be included in log output.
+        """
+        with self._timer(self.increment_put, log=log, msg=msg):
+            yield

{lsst_daf_butler-29.0.1rc1 → lsst_daf_butler-29.1.0rc2}/python/lsst/daf/butler/_config.py

@@ -1254,7 +1254,7 @@ class ConfigSubset(Config):
 
         Global defaults, at lowest priority, are found in the ``config``
         directory of the butler source tree. Additional defaults can be
-        defined using the environment variable ``$DAF_BUTLER_CONFIG_PATHS``
+        defined using the environment variable ``$DAF_BUTLER_CONFIG_PATH``
         which is a PATH-like variable where paths at the front of the list
         have priority over those later.