evefile 0.1.0rc1__py3-none-any.whl → 0.2.0__py3-none-any.whl

evefile/controllers/timestamp_mapping.py

@@ -1,30 +1,10 @@
 """
 *Mapping time stamps of monitors to position counts.*
 
-.. admonition:: Points to discuss further (without claiming to be complete)
-
-    * Mapping MonitorData to MeasureData
-
-      Monitor data (with time in milliseconds as primary axis) need to be
-      mapped to measured data (with position counts as primary axis).
-      Mapping position counts to time stamps is trivial (lookup), but *vice
-      versa* is not unique and the algorithm generally needs to be decided
-      upon. There is an age-long discussion on this topic (`#5295 note 3
-      <https://redmine.ahf.ptb.de/issues/5295#note-3>`_). For a current
-      discussion see `#7722 <https://redmine.ahf.ptb.de/issues/7722>`_.
-
-      Besides the question of how to best map one to the other (that needs to
-      be discussed, decided, clearly documented and communicated,
-      and eventually implemented): This mapping should most probably take
-      place in the controllers technical layer of the measurement functional
-      layer. The individual :class:`MonitorData
-      <evefile.entities.data.MonitorData>` class cannot do the
-      mapping without having access to the mapping table.
-
-
-For a detailed discussion/summary of the current state of affairs regarding
-the algorithm and its specification, see `#7722
-<https://redmine.ahf.ptb.de/issues/7722>`_. In short:
+Monitor data (with time in milliseconds as primary axis) need to be mapped
+to measured data (with position counts as primary axis). Mapping position
+counts to time stamps is trivial (lookup), but *vice versa* is not unique
+and the algorithm generally needs to be decided upon. In short:
 
 * Monitors corresponding to motor axes should be mapped to the *next* position.
 * Monitors corresponding to detector channels should be mapped to the
@@ -40,29 +20,13 @@ event could have occurred in the readout phase of the detectors (the
 position is incremented after moving the axes and before triggering the
 detector readout and start of nested scan modules).
 
-The :class:`TimestampData <evefile.entities.data.TimestampData>`
-class got a method :meth:`get_position()
-<evefile.entities.data.TimestampData.get_position>` to return
-position counts for given timestamps. Currently, the idea is to have one
-method handling both, scalars and lists/arrays of values, returning the same
-data type, respectively.
-
-This means that for a given :obj:`EveFile
-<evefile.boundaries.evefile.EveFile>` object, the controller carrying out
-the mapping knows to ask the :obj:`TimestampData
-<evefile.entities.data.TimestampData>` object via its :meth:`get_position()
-<evefile.entities.data.TimestampData.get_position>` method for the position
-counts corresponding to a given timestamp.
-
-Special cases that need to be addressed either here or during import of the
-data of a monitor:
+Special cases that are addressed during mapping:
 
 * Multiple values with timestamp ``-1``, *i.e.* *before* the scan has been
   started.
 
-  Probably the best solution here would be to skip all values except of the
-  last (newest) with the special timestamp ``-1``. See `#7688, note 10
-  <https://redmine.ahf.ptb.de/issues/7688#note-10>`_ for details.
+  All values except of the last (newest) with the special timestamp ``-1``
+  will be skipped for now.
 
   For future developments of the measurement engine, it may be sensible to
   record timestamps for the monitor data in actual timestamps rather than
@@ -86,16 +50,138 @@ data of a monitor:
   array.
 
 Furthermore, a requirement is that the original monitor data are retained
-when converting timestamps to position counts. This most probably means to
-create a new :obj:`MeasureData <evefile.entities.data.MeasureData>` object.
-This is the case for the additional :obj:`DeviceData
-<evefile.entities.data.DeviceData>` class as subclass of :obj:`MeasureData
-<evefile.entities.data.MeasureData>`. The next question: Where to place
-these new objects in the :class:`EveFile <evefile.boundaries.evefile.EveFile>`
-(facade) class?
+when converting timestamps to position counts. Hence, a new :obj:`MeasureData
+<evefile.entities.data.MeasureData>` object is created upon mapping.
 
 
 Module documentation
 ====================
 
 """
+
+import copy
+
+import numpy as np
+
+import evefile.entities.data
+
+
+class Mapper:
+    """
+    Map monitor datasets with timestamps to datasets with position counts.
+
+    The eve measurement program has two different concepts for recorded
+    data: datasets with position counts as primary axes (generally,
+    motor axes and detector channels), and datasets for monitors that
+    observe (monitor) values for changes and only record values upon a
+    change. The latter have timestamps (in milliseconds since start of the
+    scan) als primary axis.
+
+    To be able to correlate monitors to position counts as primary axis,
+    the datasets need to be mapped accordingly. The result of mapping a
+    monitor dataset (a :obj:`evefile.entities.data.MonitorData` object) is a
+    (new) :obj:`evefile.entities.data.DeviceData` object with the same
+    data and metadata, but position counts instead of timestamps.
+
+
+    Attributes
+    ----------
+    file : :class:`evefile.boundaries.evefile.EveFile`
+        EveFile object the mapping should be performed for.
+
+        Although mapping is carried out for individual monitors contained in
+        the EveFile object, additional information from the EveFile object
+        is necessary to perform the task.
+
+    Parameters
+    ----------
+    file : :class:`evefile.boundaries.evefile.EveFile`
+        EveFile object the mapping should be performed for.
+
+
+    Examples
+    --------
+    Usually, mapping of monitor datasets to device datasets takes place from
+    within the :class:`evefile.boundaries.evefile.EveFile` class.
+
+    To map a monitor dataset (with ID ``DetP5000:gw2370700.STAT``) that is
+    contained in the :attr:`monitors
+    <evefile.boundaries.evefile.EveFile.monitors>` attribute of the
+    :obj:`EveFile <evefile.boundaries.evefile.EveFile>` object referenced
+    here with the variable ``evefile``, perform these steps:
+
+    .. code-block::
+
+        mapper = Mapper(file=evefile)
+        device_data = mapper.map("DetP5000:gw2370700.STAT")
+
+    This will return a device dataset in the variable ``device_data`` with
+    timestamps mapped to position counts. See the :meth:`map` method for
+    further details of the actual mapping.
+
+    """
+
+    def __init__(self, file=None):
+        self.file = file
+
+    def map(self, monitor=None):
+        """
+        Map monitor dataset to device data dataset.
+
+        The device data dataset returned contains mapped position counts
+        instead of the original timestamps (in milliseconds after start of
+        the scan).
+
+        .. note::
+
+            For duplicate positions, typically values recorded before the
+            start of the actual scan and hence with the "special" timestamp
+            ``-1``, only the last position (and corresponding value) is taken.
+
+
+        Parameters
+        ----------
+        monitor : :class:`str`
+            ID of the monitor dataset to map
+
+            Note that monitors do *not* have unique names. Hence, you need
+            to provide the (unique) ID rather than the name.
+
+        Returns
+        -------
+        device_data : :class:`evefile.entities.data.DeviceData`
+            Device data with mapped position counts instead of timestamps
+
+        Raises
+        ------
+        ValueError
+            Raised if no evefile is present
+        ValueError
+            Raised if no monitor is provided
+
+        """
+        if not self.file:
+            raise ValueError("Need an evefile to map data.")
+        if not monitor:
+            raise ValueError("Need monitor to map timestamps to positions.")
+        monitor_data = self.file.monitors[monitor]
+        # Need to force load data before mapping
+        monitor_data.get_data()
+        device_data = evefile.entities.data.DeviceData()
+        device_data.metadata.copy_attributes_from(monitor_data.metadata)
+        # Take only second of each duplicate value
+        indices = np.where(
+            np.diff(
+                [
+                    *monitor_data.milliseconds,
+                    monitor_data.milliseconds[-1] + 1,
+                ]
+            )
+        )
+        device_data.position_counts = (
+            self.file.position_timestamps.get_position(
+                monitor_data.milliseconds[indices]
+            )
+        )
+        device_data.data = copy.copy(monitor_data.data[indices])
+        return device_data

evefile/controllers/version_mapping.py

@@ -188,6 +188,9 @@ Module documentation
 import datetime
 import logging
 import sys
+from collections.abc import Iterable
+
+import numpy as np
 
 from evefile import entities
 
@@ -466,7 +469,11 @@ class VersionMapper:
         importer.source = dataset.filename
         importer.item = dataset.name
         for key, value in mapping.items():
-            importer.mapping[dataset.dtype.names[key]] = value
+            # Note: datasets in array group have no column name(s)
+            if dataset.dtype.names:
+                importer.mapping[dataset.dtype.names[key]] = value
+            else:
+                importer.mapping[key] = value
         return importer
 
     @staticmethod
@@ -532,6 +539,7 @@ class VersionMapper:
         #       mapped dataset is removed from this list.
         self._map_timestamp_dataset()
         self._map_monitor_datasets()
+        self._map_array_datasets()
         self._map_axis_datasets()
         self._map_0d_datasets()
         self._map_snapshot_datasets()
@@ -559,6 +567,26 @@ class VersionMapper:
                 dataset
             )
 
+    def _map_array_datasets(self):
+        mapped_datasets = []
+        for name in self.datasets2map_in_main:
+            item = getattr(self._main_group, name)
+            # noinspection PyUnresolvedReferences
+            if isinstance(item, Iterable) and "DeviceType" in item.attributes:
+                # noinspection PyTypeChecker
+                # TODO: Distinguish between MCA and other array detectors
+                self._map_mca_dataset(hdf5_group=item)
+                # noinspection PyTypeChecker
+                mapped_datasets.append(self.get_dataset_name(item))
+        for item in mapped_datasets:
+            self.datasets2map_in_main.remove(item)
+
+    def _map_array_dataset(self, hdf5_group=None):
+        pass
+
+    def _map_mca_dataset(self, hdf5_group=None):
+        pass
+
     def _map_axis_datasets(self):
         mapped_datasets = []
         for name in self.datasets2map_in_main:
@@ -757,6 +785,158 @@ class VersionMapperV5(VersionMapper):
         dataset.metadata.unit = timestampdata.attributes["Unit"]
         self.destination.position_timestamps = dataset
 
+    def _map_mca_dataset(self, hdf5_group=None):
+        # TODO: Move up to VersionMapperV2 (at least the earliest one)
+        dataset = entities.data.MCAChannelData()
+        self.set_basic_metadata(hdf5_item=hdf5_group, dataset=dataset)
+        self._mca_dataset_set_data(dataset=dataset, hdf5_group=hdf5_group)
+        self._mca_dataset_set_options_in_main(dataset=dataset)
+        self._mca_dataset_set_options_in_snapshot(dataset=dataset)
+        self.destination.data[self.get_dataset_name(hdf5_group)] = dataset
+
+    def _mca_dataset_set_data(self, dataset=None, hdf5_group=None):
+        # Create positions vector and add it (needs to be done here)
+        positions = [int(i) for i in hdf5_group.item_names()]
+        sorter = np.argsort(positions)
+        dataset.position_counts = np.asarray(np.sort(positions), dtype="i4")
+        # Create and add importers for each individual array
+        importer_list = []
+        for position in hdf5_group:
+            importer_mapping = {
+                0: "data",
+            }
+            importer = self.get_hdf5_dataset_importer(
+                dataset=position, mapping=importer_mapping
+            )
+            importer_list.append(importer)
+        for idx in sorter:
+            dataset.importer.append(importer_list[idx])
+
+    def _mca_dataset_set_options_in_main(self, dataset=None):
+        # Handle options in main section
+        pv_base = dataset.metadata.pv.split(".")[0]
+        options_in_main = [
+            item
+            for item in self.datasets2map_in_main
+            if item.startswith(f"{pv_base}.")
+        ]
+        options_in_main.sort()
+        for option in options_in_main:
+            mapping_table = {
+                "ELTM": "life_time",
+                "ERTM": "real_time",
+            }
+            attribute = option.split(".")[-1]
+            if attribute in mapping_table:
+                importer_mapping = {1: mapping_table[attribute]}
+                importer = self.get_hdf5_dataset_importer(
+                    dataset=getattr(self.source.c1.main, option),
+                    mapping=importer_mapping,
+                )
+                dataset.importer.append(importer)
+                self.datasets2map_in_main.remove(option)
+            if attribute.startswith("R"):
+                roi = entities.data.MCAChannelROIData()
+                importer_mapping = {
+                    0: "position_counts",
+                    1: "data",
+                }
+                importer = self.get_hdf5_dataset_importer(
+                    dataset=getattr(self.source.c1.main, option),
+                    mapping=importer_mapping,
+                )
+                roi.importer.append(importer)
+                self.set_basic_metadata(
+                    hdf5_item=getattr(self.source.c1.main, option),
+                    dataset=roi,
+                )
+                dataset.roi.append(roi)
+                self.datasets2map_in_main.remove(option)
+
+    def _mca_dataset_set_options_in_snapshot(self, dataset):
+        # Handle options in snapshot section
+        pv_base = dataset.metadata.pv.split(".")[0]
+        options_in_snapshot = [
+            item
+            for item in self.datasets2map_in_snapshot
+            if item.startswith(f"{pv_base}.")
+        ]
+        options_in_snapshot.sort()
+        calibration_options = [
+            item.split(".")[-1]
+            for item in options_in_snapshot
+            if item.split(".")[-1].startswith("CAL")
+        ]
+        if calibration_options:
+            mapping_table = {
+                "CALO": "offset",
+                "CALQ": "quadratic",
+                "CALS": "slope",
+            }
+            calibration = entities.metadata.MCAChannelCalibration()
+            for option in calibration_options:
+                # HDF5 datasets are read directly and only the first data
+                # point taken from each, as calibration cannot sensibly
+                # change between scan modules of a scan.
+                name = ".".join([pv_base, option])
+                hdf5_dataset = getattr(self.source.c1.snapshot, name)
+                hdf5_dataset.get_data()
+                setattr(
+                    calibration,
+                    mapping_table[option],
+                    hdf5_dataset.data[name][0],
+                )
+                options_in_snapshot.remove(name)
+                self.datasets2map_in_snapshot.remove(name)
+            dataset.metadata.calibration = calibration
+        roi_options = [
+            item.split(".")[-1]
+            for item in options_in_snapshot
+            if item.split(".")[-1].startswith("R")
+        ]
+        if roi_options:
+            n_rois = len(set(int(item[1:-2]) for item in roi_options))
+            for idx in range(n_rois):
+                if len(dataset.roi) - 1 < idx:
+                    roi = entities.data.MCAChannelROIData()
+                    dataset.roi.append(roi)
+                else:
+                    roi = dataset.roi[idx]
+                name = ".".join([pv_base, f"R{idx}LO"])
+                hdf5_dataset = getattr(self.source.c1.snapshot, name)
+                hdf5_dataset.get_data()
+                roi.marker[0] = hdf5_dataset.data[name][0]
+                name = ".".join([pv_base, f"R{idx}HI"])
+                hdf5_dataset = getattr(self.source.c1.snapshot, name)
+                hdf5_dataset.get_data()
+                roi.marker[1] = hdf5_dataset.data[name][0]
+                name = ".".join([pv_base, f"R{idx}NM"])
+                hdf5_dataset = getattr(self.source.c1.snapshot, name)
+                hdf5_dataset.get_data()
+                roi.label = hdf5_dataset.data[name][0].decode()
+            for option in roi_options:
+                name = ".".join([pv_base, option])
+                options_in_snapshot.remove(name)
+                self.datasets2map_in_snapshot.remove(name)
+        mapping_table = {
+            "PRTM": "preset_real_time",
+            "PLTM": "preset_life_time",
+        }
+        for option, attribute in mapping_table.items():
+            name = ".".join([pv_base, option])
+            hdf5_dataset = getattr(self.source.c1.snapshot, name)
+            hdf5_dataset.get_data()
+            setattr(
+                dataset.metadata,
+                attribute,
+                hdf5_dataset.data[name][0],
+            )
+            options_in_snapshot.remove(name)
+            self.datasets2map_in_snapshot.remove(name)
+        for option in options_in_snapshot:
+            logger.warning("Option %s unmapped", option.split(".")[-1])
+            self.datasets2map_in_snapshot.remove(option)
+
     def _map_log_messages(self):
         if not hasattr(self.source, "LiveComment"):
             return