mx-bluesky 1.5.3__tar.gz → 1.5.5__tar.gz

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/.github/workflows/_helm.yml

@@ -6,6 +6,13 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
+
+      - name: checkout repo
+        uses: actions/checkout@v4
+        with:
+          # Need this to get version number from last tag
+          fetch-depth: 0
+
       - name: Install Helm
         uses: Azure/setup-helm@v4.3.0
         id: install

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/.github/workflows/_test.yml

@@ -53,7 +53,7 @@ jobs:
       - name: Run tests
         # Longer timeout for CI due to likelihood of cold caches,
         # cloud infra hw contention etc.
-        run: tox -e tests -- -m "not dlstbx" --timeout=2
+        run: tox -e tests -- -m "not dlstbx" --timeout=2 --cov=mx_bluesky --cov-report term --cov-report xml:cov.xml
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mx-bluesky
-Version: 1.5.3
+Version: 1.5.5
 Summary: Bluesky tools for MX Beamlines at DLS
 Author-email: Dominic Oram <dominic.oram@diamond.ac.uk>
 License:                                  Apache License
@@ -241,7 +241,7 @@ Requires-Dist: blueapi>=0.15.0
 Requires-Dist: ophyd>=1.10.5
 Requires-Dist: ophyd-async>=0.10.0a2
 Requires-Dist: bluesky>=1.13.1
-Requires-Dist: dls-dodal>=1.55.0
+Requires-Dist: dls-dodal==1.57.0
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: build; extra == "dev"

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/general/how-to/deploy-a-release.rst

@@ -38,7 +38,7 @@ For building and deploying a Docker image please see :doc:`../../hyperion/deploy
 
 .. note::
 
-    On i03 the installation will succeed with error messages due to RedHat7 versions of a dependency being unavailable.
+    On i03 the installation will succeed with error messages due to RedHat7 versions of dependencies being unavailable.
     This results in the installation being incomplete, thus requiring the following post-installation steps:
 
     First, on a RedHat8 workstation, run
@@ -47,6 +47,7 @@ For building and deploying a Docker image please see :doc:`../../hyperion/deploy
 
         . ./.venv/bin/activate
         pip install confluent-kafka
+        pip install contourpy
 
     Then, on the control machine, run
 

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/general/how-to/dev-ops/run-tests.rst

@@ -6,7 +6,7 @@ like tests`_, and run them to check for errors. You can run it with::
 
     $ tox -e tests
 
-It will also report coverage to the commandline and to ``cov.xml``.
+When the tests are run in GitHub CI it will also report coverage to ``codecov.io``.
 
 .. _pytest: https://pytest.org/
 .. _look like tests: https://docs.pytest.org/explanation/goodpractices.html#test-discovery

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/hyperion/reference/readme.md

@@ -1,17 +1,18 @@
-# hyperion
+# Hyperion Overview
 
 Repository for the Hyperion project to implement Unattended Data Collections on the Diamond MX beamlines using the [BlueSky](https://nsls-ii.github.io/bluesky/) / Ophyd framework from BNL.
 
-Currently the software is able to:
+The software is able to:
 
+- Fetch and decode UDC instructions from Agamemnon
+- Mount/unmount samples
+- Change energy of the beamline
 - Centre a sample, first using an optical camera, then using an xray grid scan. This centring is done at two orthogonal angles so that the sample is centred in 3D.
 - Perform a rotation scan to take diffraction data of the sample
 
 Left to do is:
 
-- Mount/unmount samples
 - Set up the beamline to be in a standard state for collection
-- Change energy of the beamline
 
 # Development Installation
 
@@ -41,8 +42,6 @@ If in a dev environment Hyperion will log to a local graylog instance instead an
 
 This uses the generic defaults for a local graylog instance. It can be accessed on `localhost:9000` where the username and password for the graylog portal are both admin.
 
-The hyperion python module can also be run directly without the startup script. It takes the same command line options, including:
-
 ## Testing
 
 Unit tests can be run with `pytest --random-order`. To see log output from tests you can turn on logging with the `--logging` command line option and then use the `-s` command line option to print logs into the console. So to run the unit tests such that all logs are at printed to the terminal, you can use `python -m pytest --random-order --logging -s`. Note that this will likely overrun your terminal buffer, so you can narrow the selection of tests with the `-k "<test name pattern>"` option.
@@ -53,6 +52,11 @@ To fake interaction and processing with Zocalo, you can run `fake_zocalo/dls_sta
 
 Tracing information (the time taken to complete different steps of experiments) is collected by an [OpenTelemetry](https://opentelemetry.io/) tracer, and currently we export this information to a local Jaeger monitor (if available). To see the tracing output, run the [Jaeger all-in-one container](https://www.jaegertracing.io/docs/1.6/getting-started/), and go to the web interface at http://localhost:16686.
 
+
+# REST API (GDA Mode)
+
+When running in GDA mode, Hyperion offers the following API
+
 ## Starting a scan
 
 To start a scan you can do the following:
@@ -86,3 +90,7 @@ To make the app write the `DEBUG` level logs stored in the `CircularMemoryHandle
 curl -X PUT http://127.0.0.1:5005/flush_debug_log
 
 ```
+
+# REST API (UDC Mode)
+
+In UDC Mode, the only endpoint available is the `/status` endpoint.

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/murko-integration/explanations/architecture.rst

@@ -15,4 +15,6 @@ It will then trigger the ``OAVToRedisForwarder`` device in ``dodal`` that will s
 
 The image streaming must be done with an ophyd device as there is too much data for it all to be emitted in bluesky documents.
 
-When the data is entered into redis it will publish a message to the redis ``murko`` channel. This will get picked up by the `socket_handler <https://github.com/DiamondLightSource/mx_auto_mjpeg_capture/tree/main/socket_handler>`_, which will forward the data to murko. Currently, during testing the socket_handler is just manually run on a workstation, `#146 <https://github.com/DiamondLightSource/mx-bluesky/issues/146>`_ should fix this.
+When the data is entered into redis it will publish a message to the redis ``murko`` channel. This will get picked up by the `redis_to_murko_forwarder <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/beamlines/i04/redis_to_murko_forwarder.py>`_, which will forward the data to murko.
+
+Murko will then enter the results back into redis where they are retrieved by the `MurkoResultsDevice <https://github.com/DiamondLightSource/dodal/blob/main/src/dodal/devices/i04/murko_results.py>`_ in ``dodal``. This device uses these results to calculate where the sample should be moved to and carry out these movements.

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/serial-crystallography-on-i24/how-to/pmac-docs.rst

@@ -55,7 +55,7 @@ in a couple of different ways.
 
    -  The ``hmz`` strings are homing commands which will reset the
       encoders counts to 0 for the axis. All three motors are homed by
-      sending the string: ``#1hmz#2hmz#3hmz``. In the plans this is done
+      sending the string: ``#5hmz#6hmz#7hmz``. In the plans this is done
       by triggering the home move:
 
       .. code:: python
@@ -74,7 +74,7 @@ in a couple of different ways.
 
 
 2. The stage motors can also be moved directly through the existing PVs
-   ``ME14E-MO-CHIP-01:{X,Y,Z}``, for example:
+   ``BL24I-EA-CHIP-01:{X,Y,Z}``, for example:
 
    .. code:: python
 

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/developer/serial-crystallography-on-i24/how-to/run-a-collection.rst

@@ -24,7 +24,7 @@ detector name from the list and clicking the ``Move Stage`` button,
 which will move the detector stage to the correct height and update the
 relative PVs.
 
-Detectors available for serial: Pilatus 6M or Eiger CdTe 9M.
+Detectors available for serial: Eiger CdTe 9M.
 
 Extruder (Serial Jet)
 =====================
@@ -58,9 +58,8 @@ in the ``Pump Probe`` panel.
 
 **WARNING** This setting requires a
 hardware change, as there are only 4 outputs on the zebra and they are
-all in use. When using the Eiger the Pilatus trigger cable should be
-used to trigger the light source. When using the pilatus the eiger
-trigger cable should be used.
+all in use. When using the Eiger the ex-Pilatus trigger cable should be
+used to trigger the light source.
 
 III - **Run collection**
 

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/user/hyperion/alerts.rst

@@ -1,11 +1,19 @@
 Alerts
 ------
 
-Hyperion can be configured to raise  in order to notify staff of issues with data collections or with Hyperion 
+Hyperion can be configured to raise alerts in order to notify staff of issues with data collections or with Hyperion 
 itself.
 
 The currently supported alerting backend uses graylog alerting to send email alert notifications.
 
+The currently supported events that will generate alerts are:
+
+- On encountering a beamline error that requires user intervention.
+- When Hyperion starts UDC collection.
+- When Hyperion finishes UDC collection (there are no more Agamemnon instructions).
+- When Hyperion releases the baton.
+- When Hyperion moves on to a new container (puck). 
+
 Graylog Alert Configuration
 ===========================
 
@@ -54,6 +62,12 @@ expand to the event timestamp.
 
 All the fields that are available in the event are available under the ``event.fields`` object, e.g. ``${event.fields.alert_summary}``
 
+The email subject and body use the JMTE templating engine, to generate more complex templates and for more 
+information see the `JMTE Project Documentation`_
+
+
+.. _JMTE Project Documentation: https://github.com/DJCordhose/jmte
+
 Event Definitions
 -----------------
 

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/user/hyperion/configuration.rst

@@ -22,4 +22,4 @@ Configuration of several properties that control Hyperion execution are availabl
     "gda.mx.hyperion.use_panda_for_gridscans", "boolean", "If true then the Panda is used instead of the Zebra for XRC gridscans" 
     "gda.mx.hyperion.xrc.use_gpu_results", "boolean", "If true, then zocalo gridscan processing uses the GPU results"
     "gda.mx.hyperion.xrc.use_roi_mode", "boolean", "If true then ROI mode is used."
-    "gda.mx.udc.hyperion.enable", "boolean",  "Enables baton passing to Hyperion."
+    "gda.mx.udc.hyperion.enable", "boolean",  "Enables Hyperion UDC mode."

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/docs/user/hyperion/index.rst

@@ -10,11 +10,13 @@ What is Hyperion?
 Hyperion is a service for running high throughput unattended data collection (UDC). It does not provide a user 
 interface, instead instructions are pulled from Agamemnon which is controlled by information obtained in ISPyB.
 
-Running Hyperion
-----------------
+The software supports two modes of operation:
 
-When installed, Hyperion should be running automatically. If it is not running, it can be (re)started from GDA by 
-invoking ``hyperion_restart()`` from the Jython console.
+* UDC mode (experimental) where Hyperion automatically fetches instructions from Agamemnon.
+* GDA mode (where GDA fetches and decodes the Agamemnon
+  instructions). GDA mode will be removed in a future release.
+
+The mode of operation is determined by configuration using the ``gda.mx.udc.hyperion.enable`` parameter
 
 
 .. toctree::

mx_bluesky-1.5.5/docs/user/hyperion/running.rst

@@ -0,0 +1,13 @@
+Running Hyperion
+----------------
+
+When installed, Hyperion should be running automatically. 
+
+Restarting Hyperion
+-------------------
+
+If Hyperion is running as a standalone server installation, it can be (re)started from GDA by 
+invoking ``hyperion_restart()`` from the Jython console.
+
+If Hyperion is running in a Kubernetes container, then it can be restarted by scaling the deployment or 
+deleting the pod (which will cause a new instance to be created)

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/pyproject.toml

@@ -48,7 +48,7 @@ dependencies = [
     "ophyd >= 1.10.5",
     "ophyd-async >= 0.10.0a2",
     "bluesky >= 1.13.1",
-    "dls-dodal >= 1.55.0",
+    "dls-dodal == 1.57.0",
 ]
 
 
@@ -144,6 +144,8 @@ filterwarnings = [
     "ignore:.*`Field` should not be instantiated.*:marshmallow.warnings.ChangedInMarshmallow4Warning",
     #Remove when we drop python 3.11
     "ignore:.*__contains__ will no longer raise TypeError.*:DeprecationWarning",
+    # epicscorelibs requires setuptools >= 70.1
+    "ignore:.*The 'wheel' package is no longer the canonical location of the 'bdist_wheel' command.*:DeprecationWarning",
 ]
 # Doctest python code in docs, python code in src docstrings, test functions in tests
 testpaths = "docs src tests/unit_tests"
@@ -177,7 +179,7 @@ allowlist_externals =
 commands =
     pre-commit: pre-commit run --all-files --show-diff-on-failure {posargs}
     type-checking: pyright src tests {posargs}
-    tests: pytest --cov=mx_bluesky --cov-report term --cov-report xml:cov.xml {posargs}
+    tests: pytest {posargs}
     docs: sphinx-{posargs:build -EW --keep-going} -T docs build/html
 commands_pre =
     docs: /usr/bin/bash -c "{toxinidir}/utility_scripts/generate_plantuml.py > \

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/run_hyperion.sh

@@ -5,6 +5,7 @@
 STOP=0
 START=1
 IN_DEV=false
+MODE=gda
 
 for option in "$@"; do
     case $option in
@@ -22,7 +23,9 @@ for option in "$@"; do
         --dev)
             IN_DEV=true
             ;;
-
+        --udc)
+            MODE=udc
+            ;;
         --help|--info|--h)
             source .venv/bin/activate
             echo "`basename $0` [options]"
@@ -38,6 +41,7 @@ Options:
   --skip-startup-connection
                           Do not connect to devices at startup
   --dev                   Enable dev mode to run from a local workspace on a development machine.
+  --udc                   Start hyperion in UDC mode instead of taking commands from GDA
   --help                  This help
 
 By default this script will start an Hyperion server unless the --no-start flag is specified.
@@ -121,7 +125,7 @@ if [[ $START == 1 ]]; then
     declare -A h_and_cb_args=( ["IN_DEV"]="$IN_DEV" )
     declare -A h_and_cb_arg_strings=( ["IN_DEV"]="--dev" )
 
-    h_commands=()
+    h_commands="--mode $MODE"
     cb_commands=()
     for i in "${!h_and_cb_args[@]}"
     do

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '1.5.3'
-__version_tuple__ = version_tuple = (1, 5, 3)
+__version__ = version = '1.5.5'
+__version_tuple__ = version_tuple = (1, 5, 5)
 
-__commit_id__ = commit_id = 'gd31a95a33'
+__commit_id__ = commit_id = 'gb6ac6ef7f'

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/beamlines/aithre_lasershaping/__init__.py

@@ -1,4 +1,5 @@
 from mx_bluesky.beamlines.aithre_lasershaping.beamline_safe import (
+    go_to_zero,
     set_beamline_safe_on_robot,
 )
 from mx_bluesky.beamlines.aithre_lasershaping.check_goniometer_performance import (
@@ -18,4 +19,5 @@ __all__ = [
     "go_to_furthest_maximum",
     "rotate_goniometer_relative",
     "jog_sample",
+    "go_to_zero",
 ]

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/beamlines/aithre_lasershaping/beamline_safe.py

@@ -34,3 +34,20 @@ def set_beamline_safe_on_robot(
         else ForceBit.NO.value
     )
     yield from bps.abs_set(robot.set_beamline_safe, set_value, wait=True)
+
+
+def go_to_zero(
+    goniometer: Goniometer = inject("goniometer"), group="move_to_zero", wait=True
+) -> MsgGenerator:
+    """
+    Rotate the goniometer and set stages to zero in preparation for robot load/unload.
+    Pass wait=False to avoid waiting for the move to complete.
+    """
+    yield from bps.abs_set(goniometer.omega, 0, group=group)
+    yield from bps.abs_set(goniometer.x, 0, group=group)
+    yield from bps.abs_set(goniometer.y, 0, group=group)
+    yield from bps.abs_set(goniometer.z, 0, group=group)
+    yield from bps.abs_set(goniometer.sampy, 0, group=group)
+    yield from bps.abs_set(goniometer.sampz, 0, group=group)
+    if wait:
+        yield from bps.wait(group=group)

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/beamlines/i04/__init__.py

@@ -1,10 +1,15 @@
 from mx_bluesky.beamlines.i04.experiment_plans.i04_grid_detect_then_xray_centre_plan import (
     i04_grid_detect_then_xray_centre,
 )
-from mx_bluesky.beamlines.i04.thawing_plan import thaw, thaw_and_stream_to_redis
+from mx_bluesky.beamlines.i04.thawing_plan import (
+    thaw,
+    thaw_and_murko_centre,
+    thaw_and_stream_to_redis,
+)
 
 __all__ = [
     "thaw",
     "thaw_and_stream_to_redis",
     "i04_grid_detect_then_xray_centre",
+    "thaw_and_murko_centre",
 ]

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/beamlines/i04/experiment_plans/i04_grid_detect_then_xray_centre_plan.py

@@ -4,7 +4,6 @@ from functools import partial
 
 import bluesky.plan_stubs as bps
 import bluesky.preprocessors as bpp
-from blueapi.core import BlueskyContext
 from bluesky.utils import MsgGenerator
 from dodal.common import inject
 from dodal.devices.aperturescatterguard import ApertureScatterguard
@@ -66,7 +65,6 @@ from mx_bluesky.common.parameters.gridscan import GridCommon, SpecifiedThreeDGri
 from mx_bluesky.common.preprocessors.preprocessors import (
     transmission_and_xbpm_feedback_for_collection_decorator,
 )
-from mx_bluesky.common.utils.context import device_composite_from_context
 from mx_bluesky.common.utils.log import LOGGER
 from mx_bluesky.phase1_zebra.device_setup_plans.setup_zebra import (
     setup_zebra_for_gridscan,
@@ -74,12 +72,6 @@ from mx_bluesky.phase1_zebra.device_setup_plans.setup_zebra import (
 )
 
 
-def create_devices(
-    context: BlueskyContext,
-) -> GridDetectThenXRayCentreComposite:
-    return device_composite_from_context(context, GridDetectThenXRayCentreComposite)
-
-
 # See https://github.com/DiamondLightSource/blueapi/issues/506 for using device composites
 def i04_grid_detect_then_xray_centre(
     parameters: GridCommon,

{mx_bluesky-1.5.3 → mx_bluesky-1.5.5}/src/mx_bluesky/beamlines/i04/redis_to_murko_forwarder.py

@@ -7,6 +7,7 @@ from typing import TypedDict
 import numpy as np
 import zmq
 from dodal.devices.i04.constants import RedisConstants
+from dodal.devices.i04.murko_results import FullMurkoResults, MurkoResult
 from numpy.typing import NDArray
 from PIL import Image
 from redis import StrictRedis
@@ -15,9 +16,6 @@ from mx_bluesky.common.utils.log import LOGGER
 
 MURKO_ADDRESS = "tcp://i04-murko-prod.diamond.ac.uk:8008"
 
-MurkoResult = dict
-FullMurkoResults = dict[str, list[MurkoResult]]
-
 
 class MurkoRequest(TypedDict):
     """See https://github.com/MartinSavko/murko#usage for more information."""
@@ -96,6 +94,7 @@ class BatchMurkoForwarder:
             ],
             "prefix": uuids,
         }
+
         results = send_to_murko_and_get_results(request_arguments)
         results_with_uuids = _correlate_results_to_uuids(request_arguments, results)
         self._send_murko_results_to_redis(sample_id, results_with_uuids)