mx-bluesky 1.4.8__tar.gz → 1.5.0__tar.gz

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/.github/workflows/_container.yml

@@ -30,14 +30,25 @@ jobs:
           DOCKER_BUILD_RECORD_UPLOAD: false
         with:
           context: .
-          file: Dockerfile.release
+          file: Dockerfile.hyperion
+          load: true
+
+      - name: Build and export Dockerfile.blueapi to local cache
+        uses: docker/build-push-action@v6
+        env:
+          DOCKER_BUILD_RECORD_UPLOAD: false
+        with:
+          context: .
+          file: Dockerfile.blueapi
           load: true
 
       - name: Create tags for publishing image
         id: meta
         uses: docker/metadata-action@v5
         with:
-          images: ghcr.io/${{ github.repository }}
+          images: |
+            ghcr.io/${{ github.repository }}
+            ghcr.io/${{ github.repository }}-blueapi
           tags: |
             type=ref,event=tag
             type=raw,value=latest
@@ -51,7 +62,19 @@ jobs:
         # Docker cache and publish it
         with:
           context: .
-          file: Dockerfile.release
+          file: Dockerfile.hyperion
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+
+      - name: Push Dockerfile.blueapi image to container registry
+        if: github.ref_type == 'tag'
+        uses: docker/build-push-action@v6
+        env:
+          DOCKER_BUILD_RECORD_UPLOAD: false
+        with:
+          context: .
+          file: Dockerfile.blueapi
           push: true
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

mx_bluesky-1.5.0/Dockerfile.blueapi

@@ -0,0 +1,14 @@
+ # See https://github.com/DiamondLightSource/mx-bluesky/issues/1004
+FROM ghcr.io/diamondlightsource/blueapi:0.10.1
+
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN apt update -y && apt upgrade -y
+# Needed for python cv2 dependency
+RUN apt install -y libgl1 libglib2.0-0
+
+RUN /venv/bin/pip install mx-bluesky
+
+# Needed so that Blueapi's setup-scratch stage has permission to modify dependencies installed 
+# in the above pip installation
+RUN mkdir -p /.cache/pip; chmod -R 777 /venv /.cache/pip

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mx-bluesky
-Version: 1.4.8
+Version: 1.5.0
 Summary: Bluesky tools for MX Beamlines at DLS
 Author-email: Dominic Oram <dominic.oram@diamond.ac.uk>
 License:                                  Apache License
@@ -220,7 +220,7 @@ Requires-Dist: flask-restful
 Requires-Dist: ispyb
 Requires-Dist: jupyterlab
 Requires-Dist: matplotlib
-Requires-Dist: nexgen
+Requires-Dist: nexgen>=0.11.0
 Requires-Dist: numpy
 Requires-Dist: opencv-python
 Requires-Dist: opentelemetry-distro
@@ -234,12 +234,12 @@ Requires-Dist: scanspec
 Requires-Dist: scipy
 Requires-Dist: semver
 Requires-Dist: matplotlib
-Requires-Dist: blueapi>=0.5.0
-Requires-Dist: daq-config-server>=0.1.1
+Requires-Dist: blueapi>=0.11.1
+Requires-Dist: daq-config-server==0.1.1
 Requires-Dist: ophyd>=1.10.5
 Requires-Dist: ophyd-async>=0.10.0a2
 Requires-Dist: bluesky>=1.13.1
-Requires-Dist: dls-dodal==1.47.0
+Requires-Dist: dls-dodal==1.49.0
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: build; extra == "dev"

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/code-map/grid_detect_xrc.rst

@@ -1,9 +1,9 @@
 Grid Detect Then Xray Centre
 ============================
 
-The aim of this page is to provide a general overview of the Hyperion ``grid_detect_then_xray_centre_plan``.
+The aim of this page is to provide a general overview of the ``grid_detect_then_xray_centre_plan``.
 
-Most of the code for the ``grid_detect_then_xray_centre_plan`` plan is in ``mx-bluesky/hyperion``. Some code is located in ``mx-bluesky/common`` and `dodal <https://github.com/DiamondLightSource/dodal>`_, and plans are increasingly being generalised and moved into ``mx-bluesky/common`` where possible. Therefore, the structure of the codebase is subject to change and this document should be updated accordingly.
+The code for the ``grid_detect_then_xray_centre`` plan is now entirely ``mx-bluesky/common/`` and `dodal <https://github.com/DiamondLightSource/dodal>`_. There is a new Hyperion entry point, ``hyperion_grid_detect_then_xray_centre`` which calls this common plan, setting up Hyperion features for it. There will soon be an i04 entry point that calls the common plan in the same way.
 
 There are then a number of plans that make up the ``grid_detect_then_xray_centre_plan`` plan. Some important ones:
 
@@ -11,7 +11,7 @@ There are then a number of plans that make up the ``grid_detect_then_xray_centre
 * :ref:`flyscan_xray_centre_plan<flyscan>` - Triggers a hardware-based grid scan and moves to the X-ray centre as returned from ``zocalo``.
 * :ref:`grid_detect_then_xray_centre_plan<grid-detect-xrc>` - This top-level plan performs an :ref:`OAV grid detection <grid-detect>` then a :ref:`flyscan x-ray centre <flyscan>`.
 
-The diagram below shows all the plans that make up the ``grid_detect_then_xray_centre_plan``. The colors indicate where these plans can be found.
+The diagram below shows all the plans that make up the ``grid_detect_then_xray_centre_plan``.
 
 .. image:: grid_detect_then_xray_centre.drawio.png
 
@@ -20,21 +20,22 @@ The diagram below shows all the plans that make up the ``grid_detect_then_xray_c
 Grid Detect Then Xray Centre Plan
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py>`__ does the following, in roughly this order:
+`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/common/experiment_plans/common_grid_detect_then_xray_centre_plan.py>`__ does the following, in roughly this order:
 
 1. If called standalone, start preparing for data collection.
-2. Perform an :ref:`OAV grid detection <grid-detect>`.
-3. Convert the parameters calculated in step 2 into something we can send to the flyscan X-ray centre.
-4. Move the backlight out, set the aperture to small, and wait for the detector to finish moving.
-5. Perform a :ref:`flyscan X-ray centre <flyscan>`.
-6. Move the sample based on the results of step 5.
+2. Move the beamstop into place if it isn't already.
+3. Perform an :ref:`OAV grid detection <grid-detect>`.
+4. Convert the parameters calculated in step 2 into something we can send to the flyscan X-ray centre.
+5. Move the backlight out, set the aperture to small, and wait for the detector to finish moving.
+6. Perform a :ref:`flyscan X-ray centre <flyscan>`.
+7. Move the sample based on the results of step 5.
 
 .. _grid-detect:
 
 OAV Grid Detection
 ~~~~~~~~~~~~~~~~~~
 
-`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/hyperion/experiment_plans/oav_grid_detection_plan.py>`__ does the following, in roughly this order:
+`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/common/experiment_plans/oav_grid_detection_plan.py>`__ does the following, in roughly this order:
 
 1. Move to omega 0.
 2. Calculate the 2D grid size using the edge arrays from the OAV.
@@ -48,7 +49,7 @@ OAV Grid Detection
 Flyscan Xray Centre No Move
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/hyperion/experiment_plans/flyscan_xray_centre_plan.py>`__ does the following, in roughly this order:
+`This plan <https://github.com/DiamondLightSource/mx-bluesky/blob/main/src/mx_bluesky/common/experiment_plans/common_flyscan_xray_centre_plan.py>`__ does the following, in roughly this order:
 
 1. Move to the desired transmission (and turn off xbpm feedback).
 2. Move to omega 0.

mx_bluesky-1.5.0/docs/developer/general/explanations/containerised_mx_bluesky.rst

@@ -0,0 +1,14 @@
+Containerised mx-bluesky
+========================
+
+There are currently two images associated with this repository which are pushed on release: hyperion, and mx-bluesky-blueapi.
+
+The Hyperion image exists because Hyperion was developed before BlueAPI was production-ready, and so doesn't use BlueAPI to schedule plans. This image is only really relevant for i03, and currently isn't used in production anywhere
+
+The ``mx-bluesky-blueapi`` image exists as a minor extension of BlueAPI's image. BlueAPI's image contains the dependencies of BlueAPI, as well as the dependencies of BlueAPI, which includes dodal. When the BlueAPI service is launched, it will do a ``pip install --no deps`` of the plan repository. For MX, this means ``mx-bluesky`` gets installed without any of its dependencies. For this reason, we have created an ``mx-bluesky-blueapi`` image which installs these extra dependencies. 
+
+This image can be used with BlueAPI's original helmchart, the only change required in the ``values.yaml`` is::
+
+    image:
+        repository: ghcr.io/diamondlightsource/mx-bluesky-blueapi
+        tag: "{desired_version}"

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/general/how-to/create-a-release.rst

@@ -17,6 +17,8 @@ Release
 6. You should now manually go through each line on the release notes and read it from the perspective of a beamline scientist. It should be clear from each what the change means to the beamline and should have links to easily find further info.
 7. Publish the release
 
+NOTE FOR USING THE MX-BLUESKY-BLUEAPI IMAGE: If using this image with BlueAPI's helmchart for deployment, the version of dodal which is installed will be the version which is pinned in the BlueAPI, rather than what's in mx-bluesky's pyproject.toml. Before releasing, you should pin BlueAPI to a version which uses a dodal version which is compatible with itself AND mx-bluesky. The deployment will fail if BlueAPI has no version which works with the desired dodal version
+
 ------------------------
 Deciding release numbers
 ------------------------

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/general/index.rst

@@ -42,6 +42,7 @@ Documentation is split into four categories, and each is also accessible from li
 
             explanations/decisions
             explanations/callback_and_run_logic
+            explanations/containerised_mx_bluesky
 
         +++
 

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/hyperion/deploying-hyperion.rst

@@ -68,7 +68,17 @@ For further information, see https://github.com/containers/podman/issues/2542
 Deploying to kubernetes
 -----------------------
 
-Once the docker image is built, the image can be deployed to kubernetes using the ``deploy_hyperion_to_k8s.sh`` script
+Once the docker image is built, the image can be deployed to kubernetes using the ``deploy_mx_bluesky_app_to_k8s.sh``
+script.
+
+Secrets
+~~~~~~~
+
+The deployment requires the following secrets to be created with the corresponding zocalo rabbitmq credentials if they
+are not already present:
+
+* ``rmq-creds``
+* ``rmq-api-reader``
 
 Production deployment
 ~~~~~~~~~~~~~~~~~~~~~
@@ -77,12 +87,13 @@ Then create and deploy the helm release
 
 ::
 
-    ./utility_scripts/deploy/deploy_hyperion_to_k8s.sh --beamline=<beamline> --checkout-to-prod hyperion
+    ./utility_scripts/deploy/deploy_mx_bluesky_app_to_k8s.sh --appVersion=x.y.z --beamline=<beamline> --checkout-to-prod hyperion hyperion
 
 This will run the ``deploy_mx_bluesky.py`` script to deploy the latest hyperion to ``/dls_sw``.
 You will be prompted to log into the beamline cluster, then it will create a helm release "hyperion".
 The source folders will be mounted as bind mounts to allow the pod to pick up changes in production.
 For production these are expected to be in the normal place defined in ``values.yaml``.
+If the non-k8s deployment of hyperion has already been performed, then omit ``--checkout-to-prod``.
 
 Development deployment
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -93,7 +104,7 @@ above, you install a dev deployment to the cluster you are currently logged into
 
 ::
 
-    ./utility_scripts/deploy/deploy_hyperion_to_k8s.sh --dev --beamline=<beamline> --repository=<your image repo> hyperion-test
+    ./utility_scripts/deploy/deploy_mx_bluesky_app_to_k8s.sh --dev --beamline=<beamline> --repository=<your image repo> hyperion-test hyperion
 
 
 The dev deployment bind-mounts the current ``hyperion`` workspace and ``../dodal`` into the container so that you can

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/hyperion/index.rst

@@ -14,6 +14,7 @@ Documentation specific for the Hyperion module within MX-Bluesky
 
             reference/param-hierarchy
             reference/readme
+            reference/coordinate-systems
             deploying-hyperion
             system-tests
 

mx_bluesky-1.5.0/docs/developer/hyperion/reference/coordinate-systems.puml

@@ -0,0 +1,20 @@
+@startuml
+box "Request Coordinate Space"
+participant "GDA/Agamemnon"
+participant Hyperion
+end box
+
+boundary omega_flip
+
+box "Hardware Coordinate Space"
+participant "Ophyd Devices"
+participant "Nexus File"
+end box
+
+"GDA/Agamemnon" -> Hyperion: Hyperion Request\n(Request Coordinate Space)
+Hyperion -> "Ophyd Devices": Rotation Operations
+"Ophyd Devices" -> "Nexus File": PCAP Data 
+Hyperion -> "Nexus File": Write metadata\nRecord actual hardware coordinate system
+
+
+@enduml

mx_bluesky-1.5.0/docs/developer/hyperion/reference/coordinate-systems.rst

@@ -0,0 +1,76 @@
+Hyperion Coordinate Systems
+===========================
+
+Rotation Coordinate Systems
+---------------------------
+
+Within hyperion coordinates are expressed both in requests to Hyperion, in commands issued to hardware, and in the 
+output files sent to downstream analysis.
+
+.. uml:: coordinate-systems.puml
+
+Within requests to Hyperion, the requested rotations are defined with respect to a fixed coordinate system, that is 
+defined to be the same for all installations and the intention is that this should be the same as that defined by the
+nexus file format when all axes are in their positive orientations. Thus the same request made across different 
+deployments of Hyperion should result in the same physical motion regardless of the coordinate space conventions of 
+the underlying hardware.
+
+In the ``RotationScan`` parameter class ``scan_width_deg`` is defined to always be positive and the direction is 
+defined by ``rotation_direction`` which can be ``POSITIVE`` or ``NEGATIVE``
+
+The coordinate systems of the underlying hardware may differ from the request. At Diamond traditionally the omega 
+axis is implemented to be in the opposite direction, therefore the commands sent to the hardware are transformed and 
+the ``omega_flip`` feature flag controls this behaviour, which by default is set to ``True`` as this is 
+generally expected to be required for the beamlines here.
+
+In the Nexus output file, there is provision for describing the choice of axis orientations in the nexus file, and 
+the ``omega_flip`` parameter affects the omega axis description in ``/entry/sample/transformations/omega``. 
+This is especially important as if PCAP position capture is enabled then referenced data files will contain positions
+with respect to this axis.
+ 
+
+Gridscan Coordinate System
+--------------------------
+
+The diagram below shows how the coordinate system used in the 3D gridscan relates to the beam and sample loop. The blue arrows indicate the direction of the beam. The red grids represent the grids used during the scan.
+
+.. image:: ../../../images/176701633-b93bba62-b2e7-4740-a863-c6b388beb14f.png
+
+Motors Coordinate System
+------------------------
+
+.. include:: <isogrk1.txt>
+
+The diagram below shows the x,y,z directions of the smargon motor coordinate system. The smargon y, z axes change with the :math:`\omega`. x,y,z are measured in millimetres.
+
+
+* The pin runs from -ve x to +ve x
+* y is +ve towards the ceiling when :math:`\omega = 0`.
+* z is +ve towards the detector/away from the oav and parallel to the beam when :math:`\omega = 0`.
+* The x and y coordinates of the image are also shown, the image feed has x from left to right, y from top to bottom, with the pin to the right of the image.
+* |ohgr| rotates from the y axis at :math:`\omega = 0` (the floor to ceiling).
+
+:math:`\omega = 0`
+
+
+.. image:: ../../../images/motor-coords-omega-0.svg
+
+
+:math:`\omega = \alpha`
+
+.. image:: ../../../images/motor-coords-omega-alpha.svg
+
+
+OAV Image Coordinate System
+---------------------------
+
+In the OAV image the below coordinate system is used (with units in pixels). 
+
+.. image:: ../../../images/207297611-f67dfc25-a3a5-4ade-a1e7-28b54a136fa6.png
+
+Conversion Between OAV Image and Smargon
+----------------------------------------
+
+At :math:`\omega = 0`, :math:`x` and :math:`y` in the camera are antiparallel to :math:`x` and :math:`y` in the smargon. :math:`y` in the camera are is parallel to :math:`z` of the smargon at :math:`x` = :math:`\frac{\pi}{2}`. Then to convert from the camera to smargon coordinate systems we map
+
+:math:`x_\text{smargon} = -x, \quad y_\text{smargon} = -y\cos(\omega),\quad z_\text{smargon} = y sin(\omega).`

{mx_bluesky-1.4.8 → mx_bluesky-1.5.0}/docs/developer/hyperion/reference/readme.md

@@ -25,14 +25,17 @@ Note that because Hyperion makes heavy use of [Dodal](https://github.com/Diamond
 
 ## Starting the bluesky runner
 
-The bluesky runner is (re)started in production by GDA when it invokes `run_hyperion.sh`. 
+The bluesky runner is (re)started in production by GDA when it invokes `run_hyperion.sh`.
 
-If you wish to attempt to run from a developer machine, then (if you are able to connect all devices) 
-you may `run_hyperion.sh --dev --beamline=<beamline>`, which will give you a running instance albeit with
-read-only devices. The `--dev` flag ensures that logging will not be sent to the production Graylog/output folders.
 
 This script will determine which beamline you are on based on the `BEAMLINE` environment variable. If on a 
-beamline Hyperion will run with `INFO` level logging, sending its logs to both production graylog and to the beamline/log/bluesky/hyperion.log on the shared file system.
+beamline Hyperion will run with `INFO` level logging, sending its logs to both production graylog and to the
+beamline/log/bluesky/hyperion.log on the shared file system.
+
+If you wish to attempt to run from a developer machine, then (if you are able to connect all devices) 
+you may `run_hyperion.sh --dev --beamline=<beamline>`, which will give you a running instance albeit with
+read-only devices. The `--dev` flag ensures that logging will not be sent to the production Graylog/output folders
+and that hyperion will create mock devices.
 
 If in a dev environment Hyperion will log to a local graylog instance instead and into a file at `./tmp/dev/hyperion.log`. A local instance of graylog will need to be running for this to work correctly. To set this up and run up the containers on your local machine run the `setup_graylog.sh` script.
 
@@ -40,12 +43,6 @@ This uses the generic defaults for a local graylog instance. It can be accessed
 
 The hyperion python module can also be run directly without the startup script. It takes the same command line options, including:
 
-`INFO` level logging of the Bluesky event documents can be enabled with the flag
-
-```
-python -m hyperion --dev --verbose-event-logging
-```
-
 ## Testing
 
 Unit tests can be run with `pytest --random-order`. To see log output from tests you can turn on logging with the `--logging` command line option and then use the `-s` command line option to print logs into the console. So to run the unit tests such that all logs are at printed to the terminal, you can use `python -m pytest --random-order --logging -s`. Note that this will likely overrun your terminal buffer, so you can narrow the selection of tests with the `-k "<test name pattern>"` option.