mx-bluesky 1.5.4__tar.gz → 1.5.6__tar.gz

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/.github/workflows/_test.yml

@@ -53,7 +53,7 @@ jobs:
       - name: Run tests
         # Longer timeout for CI due to likelihood of cold caches,
         # cloud infra hw contention etc.
-        run: tox -e tests -- -m "not dlstbx" --timeout=2
+        run: tox -e tests -- -m "not dlstbx" --timeout=2 --cov=mx_bluesky --cov-report term --cov-report xml:cov.xml
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mx-bluesky
-Version: 1.5.4
+Version: 1.5.6
 Summary: Bluesky tools for MX Beamlines at DLS
 Author-email: Dominic Oram <dominic.oram@diamond.ac.uk>
 License:                                  Apache License
@@ -241,7 +241,7 @@ Requires-Dist: blueapi>=0.15.0
 Requires-Dist: ophyd>=1.10.5
 Requires-Dist: ophyd-async>=0.10.0a2
 Requires-Dist: bluesky>=1.13.1
-Requires-Dist: dls-dodal==1.56.0
+Requires-Dist: dls-dodal==1.58.0
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: build; extra == "dev"

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/developer/general/how-to/dev-ops/run-tests.rst

@@ -6,7 +6,7 @@ like tests`_, and run them to check for errors. You can run it with::
 
     $ tox -e tests
 
-It will also report coverage to the commandline and to ``cov.xml``.
+When the tests are run in GitHub CI it will also report coverage to ``codecov.io``.
 
 .. _pytest: https://pytest.org/
 .. _look like tests: https://docs.pytest.org/explanation/goodpractices.html#test-discovery

mx_bluesky-1.5.6/docs/developer/murko-integration/explanations/DrawioImages.rst

@@ -0,0 +1,13 @@
+Draw.io Images
+--------------
+
+When working with diagrams created using Draw.io, it's important to ensure that the images retain their embedded metadata. This metadata allows the diagrams to be re-uploaded to Draw.io later for editing.
+
+To preserve this metadata:
+
+- When exporting from Draw.io, use the "File > Export As > PNG (with embedded XML)" or SVG (with embedded content) option.
+- Make sure the "Include a copy of my diagram" or similar checkbox is selected during export.
+
+This ensures that the image isn't just a flat graphic—it includes the necessary data for Draw.io to reconstruct and edit the diagram later.
+
+Tip: There is a chance of losing the metadata though if not handled properly. Sharing the image through Slack for example will delete the metadata associated with the image. Instead, use something like e-mail.

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/developer/murko-integration/explanations/architecture.rst

@@ -3,7 +3,7 @@ Murko Architecture
 
 The architecture of how Murko is integrated is still in flux but as of 07/08 the following has been tried on the beamline.
 
-.. image:: ../images/murko_setup.drawio.png
+.. image:: ../images/Murko.drawio.png
 
 The mx-bluesky code is deployed into the `beamline kubernetes cluster <https://k8s-i04.diamond.ac.uk/>`_ behind a `blueAPI <https://github.com/DiamondLightSource/blueapi>`_ REST interface. Alongside this an instance of murko is also deployed to the k8s cluster.
 
@@ -15,4 +15,6 @@ It will then trigger the ``OAVToRedisForwarder`` device in ``dodal`` that will s
 
 The image streaming must be done with an ophyd device as there is too much data for it all to be emitted in bluesky documents.
 
-When the data is entered into redis it will publish a message to the redis ``murko`` channel. This will get picked up by the `socket_handler <https://github.com/DiamondLightSource/mx_auto_mjpeg_capture/tree/main/socket_handler>`_, which will forward the data to murko. Currently, during testing the socket_handler is just manually run on a workstation, `#146 <https://github.com/DiamondLightSource/mx-bluesky/issues/146>`_ should fix this.
+When the data is entered into redis it will publish a message to the redis ``murko`` channel. This will get picked up by the `redis_to_murko_forwarder <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/beamlines/i04/redis_to_murko_forwarder.py>`_, which will forward the data to murko.
+
+Murko will then enter the results back into redis where they are retrieved by the `MurkoResultsDevice <https://github.com/DiamondLightSource/dodal/blob/main/src/dodal/devices/i04/murko_results.py>`_ in ``dodal``. This device uses these results to calculate where the sample should be moved to and carry out these movements.

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/developer/murko-integration/index.rst

@@ -16,6 +16,7 @@ Its integration at DLS, using Bluesky, is still a work in progress but this docu
             :maxdepth: 1
 
             explanations/architecture
+            explanations/DrawioImages
 
         +++
 

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/developer/serial-crystallography-on-i24/how-to/run-a-collection.rst

@@ -24,7 +24,7 @@ detector name from the list and clicking the ``Move Stage`` button,
 which will move the detector stage to the correct height and update the
 relative PVs.
 
-Detectors available for serial: Pilatus 6M or Eiger CdTe 9M.
+Detectors available for serial: Eiger CdTe 9M.
 
 Extruder (Serial Jet)
 =====================
@@ -58,9 +58,8 @@ in the ``Pump Probe`` panel.
 
 **WARNING** This setting requires a
 hardware change, as there are only 4 outputs on the zebra and they are
-all in use. When using the Eiger the Pilatus trigger cable should be
-used to trigger the light source. When using the pilatus the eiger
-trigger cable should be used.
+all in use. When using the Eiger the ex-Pilatus trigger cable should be
+used to trigger the light source.
 
 III - **Run collection**
 

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/user/hyperion/alerts.rst

@@ -6,6 +6,14 @@ itself.
 
 The currently supported alerting backend uses graylog alerting to send email alert notifications.
 
+The currently supported events that will generate alerts are:
+
+- On encountering a beamline error that requires user intervention.
+- When Hyperion starts UDC collection.
+- When Hyperion finishes UDC collection (there are no more Agamemnon instructions).
+- When Hyperion releases the baton.
+- When Hyperion moves on to a new container (puck). 
+
 Graylog Alert Configuration
 ===========================
 
@@ -54,6 +62,12 @@ expand to the event timestamp.
 
 All the fields that are available in the event are available under the ``event.fields`` object, e.g. ``${event.fields.alert_summary}``
 
+The email subject and body use the JMTE templating engine, to generate more complex templates and for more 
+information see the `JMTE Project Documentation`_
+
+
+.. _JMTE Project Documentation: https://github.com/DJCordhose/jmte
+
 Event Definitions
 -----------------
 

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/user/hyperion/configuration.rst

@@ -1,6 +1,9 @@
 Configuration
 -------------
 
+Hyperion main configuration
+===========================
+
 Configuration of several properties that control Hyperion execution are available. These can be edited in the 
 ``domain.properties`` file typically found in
 
@@ -8,6 +11,8 @@ Configuration of several properties that control Hyperion execution are availabl
 
     /dls_sw/<beamline>/software/daq_configuration/domain.properties
 
+Note that making a change to these properties generally requires a restart of the GDA server.
+
 
 .. csv-table:: Configuration properties
     :widths: auto
@@ -23,3 +28,27 @@ Configuration of several properties that control Hyperion execution are availabl
     "gda.mx.hyperion.xrc.use_gpu_results", "boolean", "If true, then zocalo gridscan processing uses the GPU results"
     "gda.mx.hyperion.xrc.use_roi_mode", "boolean", "If true then ROI mode is used."
     "gda.mx.udc.hyperion.enable", "boolean",  "Enables Hyperion UDC mode."
+
+Beamline configuration/calibration files
+========================================
+
+Hyperion makes use of other beamline configuration and calibration files which are also currently stored in 
+
+::
+
+    /dls_sw/<beamline>/software/daq_configuration
+
+These are currently shared with GDA, so changes to these files will affect both applications.
+
+Config Server
+=============
+
+If the `Config Server`_ is deployed and running, Hyperion is configured to use it in preference to reading the 
+``domain.properties`` file directly. However in the event of the config server being unavailable Hyperion will fall
+back to reading it from the filesystem.
+
+Note that currently the rest of the configuration files are not read from the config server, but the intention is that 
+ultimately it will be the source of all configuration and the remainder of the files in ``daq_configuration`` will be
+moved over to it.
+
+.. _Config Server: https://github.com/DiamondLightSource/daq-config-server/

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/user/hyperion/index.rst

@@ -18,6 +18,23 @@ The software supports two modes of operation:
 
 The mode of operation is determined by configuration using the ``gda.mx.udc.hyperion.enable`` parameter
 
+Once Hyperion has received a request, either from GDA or directly from Agamemnon, it will do the following tasks:
+
+- Robot Load (if necessary)
+- Pin tip detection using the OAV
+- Xray Centring
+- A number of data collections, depending on the number of centres returned from Zocalo and the applied selection criteria
+
+During this it will generate the following outputs:
+
+- Snapshots on robot load, XRC and rotations
+- Data collections in ISPyB for the gridscans and rotations as well as entries for robot load/unload
+- Nexus files for each data collection
+- Alert notifications on loading a new container and on beamline error conditions when intervention is required.  
+
+For a more detailed breakdown of the operation, it may be helpful to browse the  `Code Map`_
+
+.. _Code Map: ../../developer/code-map 
 
 .. toctree::
     :caption: Topics

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/docs/user/hyperion/pvs.rst

@@ -10,3 +10,12 @@ CURRENT_USER.
 Hyperion will only start operating once REQUESTED_USER has been set to Hyperion, at which point it will set 
 CURRENT_USER. To request the baton from Hyperion, set REQUESTED_USER to some other value, when Hyperion completes the
 current action, it will relinquish CURRENT_USER.
+
+Commissioning Mode
+~~~~~~~~~~~~~~~~~~
+
+The baton device also has a COMMISSIONING PV, which when the PV is set to 1, indicates that commissioning mode is 
+enabled.
+
+Commissioning mode enables testing of Hyperion when there is no beam. Beamline scientists should ensure that the 
+COMMISSIONING PV is set to 0 before running Hyperion in production.

mx_bluesky-1.5.6/docs/user/hyperion/troubleshooting.rst

@@ -0,0 +1,137 @@
+Troubleshooting
+===============
+
+.. contents::
+
+Logging
+-------
+
+Hyperion logs to a number of different locations. The sections below describe the locations that are configured via 
+environment variables for a standard server install.
+
+Graylog
+~~~~~~~
+
+Graylog is the recommended way to view logs. It is used for more centralised logging which is also more easily 
+searched and archived. Log messages are sent to the `Hyperion graylog stream <https://graylog.diamond.ac.uk/streams/66264f5519ccca6d1c9e4e03/search>`_.
+
+
+Startup Log
+~~~~~~~~~~~
+
+When ``hyperion_restart()`` is called by GDA, it will log the initial console output to a log file. This log file 
+location is 
+controlled by the ``gda.logs.dir`` property and is typically ``/dls_sw/<beamline>/logs/bluesky``.
+
+Log files
+~~~~~~~~~
+
+By default, Hyperion logs to the filesystem.
+
+The log file location is controlled by the ``LOG_DIR`` environment value. Typically this can be found in 
+``/dls_sw/<beamline>/logs/bluesky``
+
+Debug Log
+~~~~~~~~~
+
+The standard logs files do not record all messages, only those at INFO level or higher, in order to keep storage 
+requirements to a minimum. 
+In the event of an error occurring, then trace-level logging for the most recent events (by default 5000) is flushed 
+to a separate set of log files. Due to the size of these files, they are stored separately from the main log files
+. These logs are located in ``/dls/tmp/<beamline>/logs/bluesky`` by default, or 
+otherwise as specified by the ``DEBUG_LOG_DIR`` environment variable. 
+
+Server Install
+--------------
+
+Install Location
+~~~~~~~~~~~~~~~~
+
+Hyperion is generally installed in the beamline software directory:
+
+``/dls_sw/<beamline>/software/bluesky``
+
+This directory contains versioned installation folders ``mx-bluesky_vx.y.z`` for each installed version. Within the 
+directory there is a symlink ``hyperion`` to the currently active version. This is generally a symlink to either 
+``hyperion_stable`` or ``hyperion_latest`` symlinks which then point to the latest stable and development releases 
+respectively.
+
+Switching Versions
+~~~~~~~~~~~~~~~~~~
+
+For example to switch from the development to the stable release simply:
+
+::
+
+    rm hyperion
+    ln -s hyperion_latest hyperion
+
+After this you will need to run ``hyperion_restart()`` in the GDA jython console to restart hyperion
+
+Verifying that Hyperion is running
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To verify that the correct version of Hyperion is running, you should be able to view the process on the control 
+server, ssh into the control server, the output of ``ps ax`` should be something like the following, showing which 
+deployment is currently running. You should see two processes, the main process and also the external callback process. 
+
+::
+
+    $ ps ax | grep hyperion
+    1181420 ?        Sl     2:36 /dls_sw/i03/software/bluesky/mx-bluesky_v1.5.0/mx-bluesky/.venv/bin/python /dls_sw/i03/software/bluesky/mx-bluesky_v1.5.0/mx-bluesky/.venv/bin/hyperion
+    1181422 ?        Sl     1:54 /dls_sw/i03/software/bluesky/mx-bluesky_v1.5.0/mx-bluesky/.venv/bin/python /dls_sw/i03/software/bluesky/mx-bluesky_v1.5.0/mx-bluesky/.venv/bin/hyperion-callbacks
+
+Kubernetes Install
+------------------
+
+If Hyperion is deployed on Kubernetes (currently experimental) then it can be managed from the beamline kubernetes 
+dashboard, e.g. 
+https://k8s-i03-dashboard.diamond.ac.uk
+
+In the beamline namespace there will be a deployment ``hyperion-deployment``, a ``hyperion-svc`` service and associated 
+pods, ingress etc. through which the current state may be observed / managed.
+
+Logging
+~~~~~~~
+
+On kubernetes deployments, the initial startup is sent to standard IO and is captured as part of the standard 
+kubernetes logging facility.
+
+The configured logging locations are defined in the ``values.yaml`` for the specific deployment Helm chart. 
+
+Known Issues
+------------
+
+Odin Errors when there are filesystem issues
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+https://github.com/DiamondLightSource/mx-bluesky/issues/1199
+
+On occasions where there are issues with the filesystem you may see errors similar to
+
+::
+
+    ophyd.utils.errors.UnknownStatusFailure: The status (Status(obj=EpicsSignalWithRBV
+    (read_pv='BL03I-EA-EIGER-01:OD:Capture_RBV', name='eiger_odin_file_writer_capture', parent='eiger_odin_file_writer',
+    value=0, timestamp=1754488753.208739, auto_monitor=False, string=False, write_pv='BL03I-EA-EIGER-01:OD:Capture',
+    limits=False, put_complete=False), done=True, success=False) & SubscriptionStatus(device=eiger_odin_meta_ready,
+    done=False, success=False)) has failed. To obtain more specific, helpful errors in the future, update the Device
+    to use set_exception(...) instead of _finished(success=False).
+
+hyperion_restart() sometimes times out
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes hyperion_restart() will time out waiting for Hyperion to start, in the Jython console you may see the 
+following
+
+::
+
+    InteractiveConsole exception: hyperion_utils.exceptions.HyperionFailedException: Hyperion failed to start, see /dls_sw/i03/logs/bluesky/start_log.log for log
+    org.python.core.PyException: hyperion_utils.exceptions.HyperionFailedException: Hyperion failed to start, see /dls_sw/i03/logs/bluesky/start_log.log for log
+	at org.python.core.PyException.doRaise(PyException.java:239)
+	at org.python.core.Py.makeException(Py.java:1654)
+	at org.python.core.Py.makeException(Py.java:1658)
+	at org.python.core.Py.makeException(Py.java:1662)
+
+However on inspection the start log will not show any errors. Hyperion running can be verified as above `Verifying 
+that Hyperion is running`_

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/pyproject.toml

@@ -48,7 +48,7 @@ dependencies = [
     "ophyd >= 1.10.5",
     "ophyd-async >= 0.10.0a2",
     "bluesky >= 1.13.1",
-    "dls-dodal == 1.56.0",
+    "dls-dodal == 1.58.0",
 ]
 
 
@@ -179,7 +179,7 @@ allowlist_externals =
 commands =
     pre-commit: pre-commit run --all-files --show-diff-on-failure {posargs}
     type-checking: pyright src tests {posargs}
-    tests: pytest --cov=mx_bluesky --cov-report term --cov-report xml:cov.xml {posargs}
+    tests: pytest {posargs}
     docs: sphinx-{posargs:build -EW --keep-going} -T docs build/html
 commands_pre =
     docs: /usr/bin/bash -c "{toxinidir}/utility_scripts/generate_plantuml.py > \

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/src/mx_bluesky/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '1.5.4'
-__version_tuple__ = version_tuple = (1, 5, 4)
+__version__ = version = '1.5.6'
+__version_tuple__ = version_tuple = (1, 5, 6)
 
-__commit_id__ = commit_id = 'gb4b359fcc'
+__commit_id__ = commit_id = 'g6a9297fcb'

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/src/mx_bluesky/beamlines/i04/__init__.py

@@ -1,10 +1,15 @@
 from mx_bluesky.beamlines.i04.experiment_plans.i04_grid_detect_then_xray_centre_plan import (
     i04_grid_detect_then_xray_centre,
 )
-from mx_bluesky.beamlines.i04.thawing_plan import thaw, thaw_and_stream_to_redis
+from mx_bluesky.beamlines.i04.thawing_plan import (
+    thaw,
+    thaw_and_murko_centre,
+    thaw_and_stream_to_redis,
+)
 
 __all__ = [
     "thaw",
     "thaw_and_stream_to_redis",
     "i04_grid_detect_then_xray_centre",
+    "thaw_and_murko_centre",
 ]

{mx_bluesky-1.5.4 → mx_bluesky-1.5.6}/src/mx_bluesky/beamlines/i04/redis_to_murko_forwarder.py

@@ -7,6 +7,7 @@ from typing import TypedDict
 import numpy as np
 import zmq
 from dodal.devices.i04.constants import RedisConstants
+from dodal.devices.i04.murko_results import FullMurkoResults, MurkoResult
 from numpy.typing import NDArray
 from PIL import Image
 from redis import StrictRedis
@@ -15,9 +16,6 @@ from mx_bluesky.common.utils.log import LOGGER
 
 MURKO_ADDRESS = "tcp://i04-murko-prod.diamond.ac.uk:8008"
 
-MurkoResult = dict
-FullMurkoResults = dict[str, list[MurkoResult]]
-
 
 class MurkoRequest(TypedDict):
     """See https://github.com/MartinSavko/murko#usage for more information."""
@@ -96,6 +94,7 @@ class BatchMurkoForwarder:
             ],
             "prefix": uuids,
         }
+
         results = send_to_murko_and_get_results(request_arguments)
         results_with_uuids = _correlate_results_to_uuids(request_arguments, results)
         self._send_murko_results_to_redis(sample_id, results_with_uuids)