auto-trainer-api 0.9.6__py3-none-any.whl

auto_trainer_api-0.9.6.dist-info/METADATA

@@ -0,0 +1,118 @@
+Metadata-Version: 2.4
+Name: auto-trainer-api
+Version: 0.9.6
+Summary: API for interfacing with the core acquisition process via platform and language agnostic message queues.
+License: AGPL-3.0-only
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.8
+Requires-Dist: pyhumps==3.8.0
+Requires-Dist: pyzmq==26.4
+Provides-Extra: telemetry
+Requires-Dist: opentelemetry-api; extra == 'telemetry'
+Requires-Dist: opentelemetry-sdk; extra == 'telemetry'
+Provides-Extra: test
+Requires-Dist: pytest==8.2.0; extra == 'test'
+Description-Content-Type: text/markdown
+
+# Autotrainer API: Python Integration
+
+The python `auto-trainer-api` module is intended to provide an efficient means to emit information that
+is needed for local or remote management of applications running locally on the device and to receive commands from
+those sources.
+
+The exposed API is intended to be agnostic to the underlying transport layer.  The current implementation uses
+ZeroMQ.  The reasons for this decision include:
+* Relatively low overhead for the acquisition application
+* Does not require either side to manage connections and know when the other side is available or changes availability
+* Does not require an additional process or service to maintain a persistent message queue (e.g., RabbitMQ)
+  * Persistent data is managed elsewhere
+
+## Client Integration
+There are two points of integration available for clients to support the remote interface.  The first allows publishing
+"events" for state and property changes that occur in the client.  The second is a "command" interface for the client
+to receive command requests from remote sources.
+
+Both interfaces are provided through an instance of the `RpcService` protocol.  An instance can be obtained via
+`create_api_service(...)` which constructs the specific concrete implementation.  After creation, the service must
+be explicitly started (`start(...)`) and can be stopped (`stop(...)`).  Once stopped, an instance can not be
+restarted.  If a connection should be reestablished after stopping an instance, a new instance should be created and
+started.
+
+### Events
+Events are supported through the `send_dict(topic: ApiTopic, message: dict)` method on the `RpcService` instance.
+
+The first argument is the appropriate `ApiTopic` value for the event.  The most common for clients to support are
+* `ApiTopic.EVENT` - the topic for most events representing general activity in the about or high-level state changes
+* `ApiTopic.EMERGENCY` - a dedicated topic for emergency-related events such as emergency-stop/resume, alarm changes, etc. [1]
+* `ApiTopic.PROPERTY_CHANGE` - lower-level property changes on specific objects of interest
+* `ApiTopic.COMMAND_RESULT` - command responses for non-immediate commands (see Commands section)
+
+The second argument is a dictionary whose contents depend on the topic.  All elements must serializable to JSON.
+
+#### ApiTopic.EVENT
+The dictionary contains the following entries:
+* `kind` - an `ApiEventKind` value
+* `when` - a wall-clock value of time
+* `index` - monotonically increasing timestamp w/units if nanoseconds
+* `context` - an object whose contents depend on the `ApiEventKind`; may be None for some event kinds
+
+#### ApiTopic.EMERGENCY
+Still under development.
+
+#### ApiTopic.PROPERTY_CHANGE
+Still under development.
+
+#### ApiTopic.COMMAND_RESULT
+An instance of `ApiCommandRequestResponse`.
+
+[1] Currently these are supported as special cases of `ApiTopic.EVENT`.  At some point these should be transitioned to
+the dedicated topic.
+
+### Commands
+Commands from external sources are supported by registering a `CommandRequestDelegate` with the `RpcService` instance.
+The delegate receives an instance of `ApiCommandRequest` and must return an instance of `ApiCommandRequestResponse`.
+
+The primary property of the `ApiCommandRequest` is `command` which is an `ApiCommand` value.  Depending on the command,
+there may also be a dictionary in the `data` property with arguments or other information relevant to the command.
+The `nonce` property can be ignored if the command is handled synchronously.  For commands that send an
+`ApiTopic.COMMAND_RESULT` event after completion (see below), the nonce must be stored to associate with that event
+(along with the `command` value).
+
+The returned `ApiCommandRequestResponse` object contains one require field
+* `result` - a value of `ApiCommandReqeustResult`
+
+There are also three optional fields
+* `data` - an optional dictionary with results from the command beyond success/failure (often will be None)
+* `error_code` an integer error code value if the command is not successful or can't be initiated
+* `error_message` an integer error code value if the command is not successful or can't be initiated
+
+The expected contents of the `data` property are defined by the command, but is typically `None`.
+
+The `error_code` property should be a non-zero value if there is an error code to report.
+
+There are two fields on the ApiCommandRequestResponse object that are ignored as part of the returned object from
+the command delegate: `command` and `nonce`.  See _Asynchronous Commands_ for when these fields are required.
+
+
+#### Asynchronous Commands
+The command delegate is expected to return "immediately" (low millisecond type of time frame).  If the command is not
+deterministically fast, it is expected to immediately return an `ApiCommandRequestResponse` with a `result` value of 
+`ApiCommandReqeustResult.PENDING_WITH_NOTIFICATION`.
+
+Once the action associated with the command is complete, the client should send an Event (previous section), with a
+topic of `ApiTopic.COMMAND_RESULT`.  The `message` argument of the `send_dict` method should be another instance of
+`ApiCommandRequestResponse`.  The `command` and `nonce` properties of the response object should be set to the values
+received in the `ApiCommandRequest` (the client is responsible for storing these values until needed).  Note that
+those two properties are ignored for synchronous command handling, but required for asynchronous responses.
+
+
+## Publishing
+`python -m build`
+
+`python -m twine upload dist/*` (requires PyPi API token)
+
+## Installation
+The package is published to the PyPi package index and can be installed with standard pip commands.
+
+`pip install auto-trainer-api`

auto_trainer_api-0.9.6.dist-info/RECORD

@@ -0,0 +1,12 @@
+autotrainer/api/__init__.py,sha256=8GUKQ8L-gGgU8XlMztL88zwcjalregEVh__Mo6GGRg4,748
+autotrainer/api/api_event_kind.py,sha256=cKwMDhlTUGY6SM_A1r2vWfHIDPRHYHtSDkWdTt9oDbs,3159
+autotrainer/api/api_options.py,sha256=LJrWqVe6JhYKw9UDL3uJcX4HTkwLW0STaNCGpPWRkoc,761
+autotrainer/api/rpc_service.py,sha256=1r7RFrSgsjfZSMunq1UgA8AEP6oLXF48waM8TC-OMI4,18628
+autotrainer/api/util.py,sha256=RfN6xMgCUahOz6RWMvSUkuw3mz0JEB2cpYoZBVdDY5M,1244
+autotrainer/api/telemtry/__init__.py,sha256=gd013s0gvD2Wc-hj2tGkvQQGNR3u0f8ZGYxbYvITWfo,57
+autotrainer/api/telemtry/open_telemetry_service.py,sha256=V-bDe_UX1I4iPtBgKV4R4tAAuGbsE3kyxGtr5ZD2mJc,2018
+autotrainer/api/zeromq/__init__.py,sha256=L0txGZRsARnVMfuKpEg9E22WkiuoU1FRmzqMAQCqtf4,50
+autotrainer/api/zeromq/zeromq_api_service.py,sha256=WiG7B7rFJNESdDZhzxkmvHY03ei5T9PXeNgRFhJUTjg,4981
+auto_trainer_api-0.9.6.dist-info/METADATA,sha256=3P0jS8siCnK0uIw9MyIjvPfVSq7sPnZV3DozRmPgs8g,6332
+auto_trainer_api-0.9.6.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+auto_trainer_api-0.9.6.dist-info/RECORD,,

auto_trainer_api-0.9.6.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.27.0
+Root-Is-Purelib: true
+Tag: py3-none-any

autotrainer/api/__init__.py

@@ -0,0 +1,25 @@
+"""
+API Service functionality for Autotrainer.
+"""
+
+from typing import Optional
+
+from .api_options import ApiOptions, create_default_api_options
+from .rpc_service import ApiTopic, ApiCommandRequest, ApiCommandRequestResponse, ApiCommandReqeustResult, RpcService
+
+from .util import patch_uuid_encoder
+
+
+def create_api_service(options: ApiOptions) -> Optional[RpcService]:
+    from .zeromq import ZeroMQApiService
+
+    # Several autotrainer messages may contain a UUID, which is not handled by the default JSON encoder.
+    # patch_uuid_encoder()
+
+    # TODO Enable when ready
+    # configure_telemetry(options.telemetry)
+
+    if options.rpc.enable:
+        return ZeroMQApiService(options.rpc)
+    else:
+        return None

autotrainer/api/api_event_kind.py

@@ -0,0 +1,120 @@
+from enum import IntEnum
+
+
+# This is currently a duplicate of what is in the auto-trainer application repository.  At some point this will be the
+# source of truth.
+class ApiEventKind(IntEnum):
+    """
+    0000-0999   Core/System
+    1000-1999   Behavior
+    2000-2999   Device
+    3000-3999   Inference
+    4000-4999   Analysis
+    5000-5999   Training
+    9000-9999   Application
+    """
+
+    # Core/System
+    emergencyStop = 101
+    emergencyResume = 102
+
+    applicationLaunched = 201
+    applicationTerminating = 202
+
+    propertyChanged = 501
+
+    # Behavior
+    algorithmPause = 1001
+    algorithmResume = 1002
+
+    tunnelEnter = 1101
+    tunnelExit = 1102
+
+    pelletLoadCan = 1201
+    pelletLoadBegin = 1202
+    pelletLoadEnd = 1203
+    pelletSendCan = 1204
+    pelletSendBegin = 1205
+    pelletSendEnd = 1206
+    pelletCoverCan = 1207
+    pelletCoverBegin = 1208
+    pelletCoverEnd = 1209
+    pelletReleaseCan = 1210
+    pelletReleaseBegin = 1211
+    pelletReleaseEnd = 1212
+    pelletHomeCan = 1213
+    pelletHomeBegin = 1214
+    pelletHomeEnd = 1215
+    pelletPrereleaseCan = 1216
+    pelletPrereleaseBegin = 1217
+    pelletPrereleaseEnd = 1218
+    pelletAcknowledgeToken = 1298
+    pelletExternalToken = 1299
+
+    sessionStarting = 1301
+    sessionStarted = 1302
+    sessionEnding = 1303
+    sessionEnded = 1304
+    sessionPelletIncrease = 1311
+    sessionPelletDecrease = 1312
+    sessionMouseSeen = 1321
+
+    dayStarted = 1401
+    dayIncreasePellet = 1411
+    dayDecreasePellet = 1412
+
+    pelletSeen = 1501
+    pelletPresented = 1502
+    pelletSuccessfulReach = 1503
+
+    triangleSeen = 1550
+
+    headfixBaselineChanged = 1601
+    headfixLoadCellChanged = 1602
+    headfixLoadCellChangedInIntersession = 1603
+    headfixLoadCellChangedWrongState = 1604
+    headfixAutoTare = 1611
+
+    autoClampIntensityChanged = 1621
+    autoClampReleaseToneFreqChanged = 1622
+    autoClampReleaseDelayChanged = 1623
+
+    intersessionSegmentationCan = 1701
+    intersessionSegmentationBegin = 1702
+    intersessionSegmentationEnd = 1703
+    intersessionSegmentationNonceMismatch = 1704
+    intersessionSegmentationError = 1705
+    intersessionSegmentationSave = 1706
+    intersessionSegmentationSaveError = 1707
+    intersessionSegmentationInputError = 1708
+    intersessionDetectionCan = 1711
+    intersessionDetectionBegin = 1712
+    intersessionDetectionEnd = 1713
+    intersessionDetectionNonceMismatch = 1714
+    intersessionDetectionError = 1715
+    intersessionDetectionSave = 1716
+    intersessionDetectionSaveError = 1717
+    intersessionShiftX = 1731
+    intersessionShiftY = 1732
+    intersessionShiftZ = 1733
+
+    headFixationForceDetectorChanged = 1801
+    headFixationEnabled = 1802
+
+    # Device
+    deviceCommandSend = 2001
+    deviceCommandAcknowledge = 2002
+
+    # Analysis
+    loadCellEngagedChanged = 4001
+    headbarPressureEngagedChanged = 4011
+
+    # Training
+    trainingModeChanged = 5001
+
+    trainingPlanLoad = 5101
+
+    trainingPhaseEnter = 5201
+    trainingPhaseExit = 5202
+
+    trainingProgressUpdate = 5501

autotrainer/api/api_options.py

@@ -0,0 +1,32 @@
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass(frozen=True)
+class TelemetryOptions:
+    enable: bool = False
+    endpoint: Optional[str] = None
+    api_key: str = ""
+
+
+@dataclass(frozen=True)
+class RpcOptions:
+    enable: bool = True
+    identifier: str = "autotrainer-device"
+    heartbeat_interval: int = 5
+    subscriber_port: int = 5556
+    command_port: int = 5557
+
+
+@dataclass(frozen=True)
+class ApiOptions:
+    rpc: Optional[RpcOptions] = None
+    telemetry: Optional[TelemetryOptions] = None
+
+
+def create_default_api_options() -> ApiOptions:
+    """
+    Create default API options for the Autotrainer API service.
+    """
+
+    return ApiOptions(rpc=RpcOptions(), telemetry=TelemetryOptions())

autotrainer/api/rpc_service.py

@@ -0,0 +1,473 @@
+import json
+import logging
+import time
+from dataclasses import dataclass, asdict
+from enum import IntEnum
+from queue import Queue, Empty
+from threading import Timer, Thread
+from typing import Optional, Protocol, Any
+from typing_extensions import Self
+
+import humps
+
+from .api_event_kind import ApiEventKind
+from .api_options import RpcOptions
+
+logger = logging.getLogger(__name__)
+
+
+class ApiTopic(IntEnum):
+    """
+    A topic is required for all published messages.  This allows subscribers to filter messages through the message
+    queue functionality rather than seeing all messages and filtering themselves.
+
+    Any 4-byte integer value is valid.
+    """
+    ANY = 0
+    HEARTBEAT = 1001
+    """System heartbeat message indicating service availability."""
+    EMERGENCY = 2001
+    """Emergency only messages."""
+    EVENT = 4001
+    """
+    Data generated from the published under the 'Event' umbrella, which is typically major system/application events.
+    """
+    PROPERTY_CHANGE = 5001
+    """
+    Data generated from the published under the 'Event' umbrella, which is typically major system/application events.
+    """
+    COMMAND_RESULT = 6001
+    """ Responses to asynchronous command handling. """
+
+
+class ApiCommand(IntEnum):
+    """
+    A command value is required for all command requests.
+    """
+    NONE = 0
+    START_ACQUISITION = 100
+    STOP_ACQUISITION = 110
+    EMERGENCY_STOP = 130
+    EMERGENCY_RESUME = 135
+    """No command.  Can be used to verify connection or that the receiver is configured to receive commands."""
+    USER_DEFINED = 99999
+    """A custom command defined between a particular commander and receiver.  Additional information can be 
+    defined in the `data` field."""
+
+    @classmethod
+    def is_member(cls, value):
+        return value in cls._value2member_map_
+
+
+@dataclass(frozen=True)
+class ApiCommandRequest:
+    """
+    A command request contains the command and any associated data.
+    """
+    command: ApiCommand
+    custom_command: int = -1
+    nonce: int = -1
+    data: Optional[dict] = None
+
+    @classmethod
+    def parse_bytes(cls, message: bytes) -> Optional[Self]:
+        obj = json.loads(humps.decamelize(message.decode("utf8")))
+
+        return ApiCommandRequest.parse_object(obj)
+
+    @classmethod
+    def parse_object(cls, obj: Any) -> Optional[Self]:
+        if "command" in obj:
+            command = obj["command"]
+            if ApiCommand.is_member(command):
+                command = ApiCommand(command)
+            else:
+                command = ApiCommand.USER_DEFINED
+            if "custom_command" in obj:
+                custom_command = obj["custom_command"]
+            else:
+                custom_command = -1
+            if "data" in obj:
+                data = obj["data"]
+            else:
+                data = None
+            if "nonce" in obj:
+                nonce = obj["nonce"]
+            else:
+                nonce = 0
+
+            return ApiCommandRequest(command=command, custom_command=custom_command, nonce=nonce, data=data)
+
+        return None
+
+
+class ApiCommandReqeustResult(IntEnum):
+    """
+    Result of a command request.
+    """
+    UNRECOGNIZED = 0
+    """The client does not support this operation."""
+    SUCCESS = 100
+    """the command executed successfully."""
+    PENDING = 200
+    """The command was initiated but may not be complete."""
+    PENDING_WITH_NOTIFICATION = 201
+    """Command was initiated.  A specific event will post when the command completes."""
+    FAILED = 400
+    """The command was attempted, but could not be completed."""
+    EXCEPTION = 500
+    """The command was attempted and generated an exception."""
+    UNAVAILABLE = 9999
+    """The command is recognized, but not supported at this time."""
+
+
+class ApiCommandRequestErrorKind(IntEnum):
+    NONE = 0
+    SYSTEM_ERROR = 0x01
+    COMMAND_ERROR = 0x02
+
+
+ApiCommandRequestSystemErrorSerialization: int = 0
+
+NotificationEventKinds = [ApiEventKind.emergencyStop, ApiEventKind.emergencyResume]
+
+
+@dataclass(frozen=True)
+class ApiCommandRequestResponse:
+    result: ApiCommandReqeustResult
+
+    data: Optional[dict] = None
+
+    error_code: int = 0
+    error_message: Optional[str] = None
+
+    command: ApiCommand = ApiCommand.NONE
+    """For synchronous command handling, this does not need to be set."""
+    nonce: int = -1
+    """For synchronous command handling, this does not need to be set."""
+
+
+@dataclass(frozen=True)
+class ApiCommandRequestServiceResponse:
+    """
+    A command response contains the command and any associated data.
+
+    If the command request will have a result asynchronously in the future, one pattern would be to return some form
+    of context in the data field that the client can use as a reference in a future published message.
+    """
+    nonce: int
+    """
+    This will be set to 0 if a command request could not be deserialized.  Caller should set the nonce to any value > 0
+    if they want to be able to identify commands that were not simple unrecognized, but unparseable.
+    """
+    command: ApiCommand
+    result: ApiCommandReqeustResult = ApiCommandReqeustResult.UNRECOGNIZED
+    data: Optional[dict] = None
+    error_kind: ApiCommandRequestErrorKind = ApiCommandRequestErrorKind.NONE
+    error_code: int = 0
+    error_message: Optional[str] = None
+
+    def as_bytes(self, allow_fallback: bool = True) -> bytes:
+        """
+        This is guaranteed to return a valid response, even if modified due to any errors in serialization.  It must
+        not throw.
+
+        :param allow_fallback: true to allow serialization without the 'data' element if serialization initially fails.
+        :return: serialized message as bytes
+        """
+        try:
+            return humps.camelize(json.dumps(self.__dict__)).encode("utf8")
+        except Exception as ex:
+            logger.error(ex)
+
+        if allow_fallback:
+            # Assume it is an issue w/the contents of the user-definable dictionary contents
+            contents = self.__dict__
+            contents["data"] = None
+            # If the next attempt works, this will have been the situation.
+            contents["error_kind"] = ApiCommandRequestErrorKind.SYSTEM_ERROR
+            contents["error_code"] = ApiCommandRequestSystemErrorSerialization
+            contents["error_message"] = "An error occurred serializing the 'data' element of the response."
+            try:
+                return humps.camelize(json.dumps(contents)).encode("utf8")
+            except Exception as ex:
+                logger.error(ex)
+
+        serialization_error = {"nonce": self.nonce, "command": self.command, "result": self.result,
+                               "error_kind": ApiCommandRequestErrorKind.SYSTEM_ERROR,
+                               "error_code": ApiCommandRequestSystemErrorSerialization,
+                               "error_message": "An error occurred serializing the response."}
+
+        return humps.camelize(json.dumps(serialization_error)).encode("utf8")
+
+    @staticmethod
+    def for_exception(command: ApiCommand, nonce: int, ex: Exception):
+        return ApiCommandRequestServiceResponse(command=command, nonce=nonce, result=ApiCommandReqeustResult.EXCEPTION,
+                                                error_message=str(ex))
+
+
+class CommandRequestDelegate(Protocol):
+    """
+    This callback is expected to be fast.  It is intended to initiate a command, not necessarily complete it.  Any
+    non-trivial action is expected to accept the command request and return, perform the action on a non-calling thread
+    or process, and use the message publishing API to report changes, results, etc.
+    """
+
+    def __call__(self, request: ApiCommandRequest) -> ApiCommandRequestResponse: ...
+
+
+class ApiMessageQueueService(Protocol):
+    """
+    Minimum requirements to fulfill the API service message queue interface.  Implementation details are left to the
+    implementation.
+
+    Implementations are required to be able to publish messages to one or more subscribers.
+    """
+
+    def send(self, topic: ApiTopic, data: bytes) -> bool: ...
+
+    def send_string(self, topic: ApiTopic, message: str) -> bool: ...
+
+    def send_dict(self, topic: ApiTopic, message: dict) -> bool: ...
+
+
+class ApiCommandRequestService(Protocol):
+    """
+    Minimum requirements to fulfill the API service command provider interface.  Implementation details are left to the
+    implementation.
+
+    Implementations are required to be able to receive command requests from one or more clients, deliver those requests
+    to a registered handler, and provide an immediate response to the requester.  The response is a response to the
+    _command request_ not necessarily the response to the command itself.  See CommandCallback for additional details.
+    """
+
+    @property
+    def command_request_delegate(self) -> Optional[CommandRequestDelegate]: ...
+
+    @command_request_delegate.setter
+    def command_request_delegate(self, value: Optional[CommandRequestDelegate]): ...
+
+
+@dataclass
+class HeartbeatMessage:
+    identifier: str
+    version: str
+    timestamp: float
+
+
+class RpcService:
+    def __init__(self, options: RpcOptions):
+        self._subscriber_port = options.subscriber_port
+        self._command_port = options.command_port
+        self._identifier = options.identifier
+        self._heartbeat_interval = options.heartbeat_interval
+        self._heartbeat_timer = None
+        self._heartbeat: HeartbeatMessage = HeartbeatMessage(
+            identifier=options.identifier,
+            version="0.9.6",
+            timestamp=0.0
+        )
+
+        self._command_callback: Optional[CommandRequestDelegate] = None
+
+        self._thread = None
+
+        self._termination_requested = False
+
+        self._queue = Queue()
+
+    @property
+    def subscriber_port(self) -> int:
+        return self._subscriber_port
+
+    @property
+    def command_port(self) -> int:
+        return self._command_port
+
+    @property
+    def identifier(self) -> str:
+        return self._identifier
+
+    @property
+    def heartbeat_interval(self) -> int:
+        return self._heartbeat_interval
+
+    @property
+    def command_request_delegate(self) -> Optional[CommandRequestDelegate]:
+        return self._command_callback
+
+    @command_request_delegate.setter
+    def command_request_delegate(self, value: Optional[CommandRequestDelegate]):
+        if not self._termination_requested:
+            self._command_callback = value
+        else:
+            logger.warning("The RPC service has been terminated. command_request_delegate not set")
+
+    def start(self):
+        if self._thread is None and not self._termination_requested:
+            self._queue = Queue()
+            self._thread = Thread(target=self._run, name="RpcServiceThread")
+            self._thread.start()
+        else:
+            raise RuntimeError("The RPC service has already been started.")
+
+    def stop(self):
+        if self._thread is not None and not self._termination_requested:
+            self._termination_requested = True
+            self._command_callback = None
+            if self._thread.is_alive():
+                self._thread.join()
+
+    def send(self, topic: ApiTopic, data: bytes) -> bool:
+        if self._queue is None:
+            raise RuntimeError("The RPC service has not been started.")
+
+        if not self._termination_requested:
+            self._queue.put(lambda: self._send(topic, data))
+            return True
+
+        logger.warning("The RPC service has been terminated.  Message not sent.")
+        return False
+
+    def send_string(self, topic: ApiTopic, message: str) -> bool:
+        if self._queue is None:
+            raise RuntimeError("The RPC service has not been started.")
+
+        if not self._termination_requested:
+            self._queue.put(lambda: self._send_string(topic, message))
+            return True
+
+        logger.warning("The RPC service has been terminated.  Message not sent.")
+        return False
+
+    def send_dict(self, topic: ApiTopic, message: dict) -> bool:
+        if self._queue is None:
+            raise RuntimeError("The RPC service has not been started.")
+
+        if not self._termination_requested:
+            if topic == ApiTopic.EVENT and "kind" in message and message["kind"] in NotificationEventKinds:
+                # If an emergency event is posted in the normal event channel, ensure it goes out on the emergency
+                # topic as well.
+                self._queue.put(lambda: self._send_dict(ApiTopic.EMERGENCY, message))
+            self._queue.put(lambda: self._send_dict(topic, message))
+
+            return True
+
+        logger.warning("The RPC service has been terminated.  Message not sent.")
+        return False
+
+    def _run(self):
+        try:
+            if not self._start():
+                logger.error(f"failed to start api service")
+                self._update_after_run()
+                return
+        except Exception:
+            logger.exception("exception starting api service")
+            self._update_after_run()
+            return
+
+        self._queue_heartbeat()
+
+        while not self._termination_requested:
+            # Required to be non-blocking and exception safe by the rpc-specific implementation.
+            request = self._get_next_command_request()
+
+            if request is not None:
+                # Expected to happen fast.  Most message queue implementations that provide a request/response-type
+                # pattern require that a response be sent before the next request is received.  And the associated
+                # client/caller implementation requires the response before accepting another request.
+                #
+                # If this becomes an untenable requirement, a different pattern will be required for command
+                # requests (and the underlying implementations update).  However, anything that allows interleaving
+                # multiple requests from the same client with responses would effectively be the same as this
+                # pattern where there is an immediate _request_ response and a delayed _command_ response through
+                # the message queue.
+                #
+                # NOTE: There is no inherent limitation in this pattern with multiple requests from multiple
+                # clients.  This is related to the handling of each individual client.
+
+                if self._command_callback is not None:
+                    try:
+                        # Can not assume the registered delegate will be well-behaved.
+                        client_response = self._command_callback(request)
+
+                        error_kind = ApiCommandRequestErrorKind.COMMAND_ERROR if client_response.error_code != 0 else ApiCommandRequestErrorKind.NONE
+
+                        self._send_command_response(
+                            ApiCommandRequestServiceResponse(request.nonce, request.command, client_response.result,
+                                                             client_response.data, error_kind,
+                                                             client_response.error_code, client_response.error_message))
+                    except Exception as e:
+                        self._send_command_response(
+                            ApiCommandRequestServiceResponse.for_exception(request.command, request.nonce, e))
+                else:
+                    # Must provide a response, even if no one that registered this service cares (using for the
+                    # message queue only, etc.).
+                    self._send_command_response(
+                        ApiCommandRequestServiceResponse(command=request.command, nonce=request.nonce,
+                                                         result=ApiCommandReqeustResult.UNAVAILABLE))
+
+            try:
+                action = self._queue.get(timeout=0.05)
+
+                # This is performed by the internal rpc implementation.  It should be exception safe or the
+                # implementation should be corrected.
+                action()
+
+            except Empty:
+                # Expected from get_nowait() if there is nothing in the queue.
+                pass
+
+        self._update_after_run()
+
+    def _update_after_run(self):
+        self._cancel_heartbeat()
+
+        self._stop()
+
+    def _queue_heartbeat(self):
+        self._heartbeat_timer = Timer(self._heartbeat_interval, self._heartbeat_timer_callback)
+        self._heartbeat_timer.start()
+
+    def _cancel_heartbeat(self):
+        if self._heartbeat_timer is not None:
+            self._heartbeat_timer.cancel()
+            self._heartbeat_timer = None
+
+    def _heartbeat_timer_callback(self):
+        # Must happen on the queue processing thread.
+        if not self._termination_requested:
+            self._heartbeat.timestamp = time.time()
+            self._send_dict(ApiTopic.HEARTBEAT, asdict(self._heartbeat))
+            self._queue_heartbeat()
+
+    def _start(self) -> bool:
+        raise NotImplementedError("Subclasses must implement _start()")
+
+    def _stop(self):
+        raise NotImplementedError("Subclasses must implement _stop()")
+
+    def _send(self, topic: ApiTopic, data: bytes) -> bool:
+        return False
+
+    def _send_string(self, topic: ApiTopic, message: str) -> bool:
+        return False
+
+    def _send_dict(self, topic: ApiTopic, message: dict) -> bool:
+        return False
+
+    def _get_next_command_request(self) -> Optional[ApiCommandRequest]:
+        """
+        If a command request is returned, the subclass/implementation can assume that a responses will be sent
+        (via the `_command_response()_` method).  If something is received by the implementation that can not be
+        returned as a valid command request (malformed, incomplete, etc.), it is the responsibility of the
+        implementation to provide a response if that is required by the particular implementation (e.g., request/reply
+        requiring a response to every request, etc.).
+
+        :return: a command request if available, None otherwise
+        """
+        return None
+
+    def _send_command_response(self, response: ApiCommandRequestServiceResponse):
+        pass

autotrainer/api/telemtry/__init__.py

@@ -0,0 +1 @@
+from .open_telemetry_service import configure_telemetry

autotrainer/api/telemtry/open_telemetry_service.py

@@ -0,0 +1,51 @@
+import importlib.util
+import logging
+import os
+from typing import Optional
+
+from ..api_options import TelemetryOptions
+
+_spec_opentelemetry = importlib.util.find_spec("opentelemetry")
+
+logger = logging.getLogger(__name__)
+
+
+def configure_telemetry(options: Optional[TelemetryOptions]) -> bool:
+    if options is None or not options.enable:
+        logger.debug(f"telemetry options {'missing' if options is None else 'disabled'}.")
+        return False
+
+    if _spec_opentelemetry is None:
+        logger.warning("telemetry enabled however a required dependency is missing.")
+        return False
+
+    endpoint = options.endpoint if options.endpoint is not None else os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
+
+    if endpoint is None:
+        logger.warning("telemetry enabled however the endpoint is not specified.")
+        return False
+
+    from opentelemetry.sdk.resources import SERVICE_NAME, Resource
+
+    from opentelemetry import trace
+    from opentelemetry.sdk.trace import TracerProvider
+    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+    from opentelemetry.sdk.trace.export import BatchSpanProcessor
+
+    from opentelemetry import metrics
+    from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
+    from opentelemetry.sdk.metrics import MeterProvider
+    from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
+
+    resource = Resource(attributes={SERVICE_NAME: "auto-trainer"})
+
+    trace_provider = TracerProvider(resource=resource)
+    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces"))
+    trace_provider.add_span_processor(processor)
+    trace.set_tracer_provider(trace_provider)
+
+    reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=f"{endpoint}/v1/metrics"))
+    meter_provider = MeterProvider(resource=resource, metric_readers=[reader])
+    metrics.set_meter_provider(meter_provider)
+
+    return True

autotrainer/api/util.py

@@ -0,0 +1,40 @@
+import socket
+from datetime import datetime
+from json import JSONEncoder
+from uuid import UUID
+
+_IS_JSON_ENCODER_PATCHED = False
+
+
+def get_ip4_addr_str() -> str:
+    # gethostname() and gethostbyname() and associated IP lookup have proven unreliable on deployed devices where the
+    # configuration is not perfect.  This method assumes access to the internet (Google DNS) which has its own
+    # limitations.  A more complex implementation to manage all conditions but avoid ending up with 12.0.0.1 when an
+    # actual address is available is needed.
+    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    try:
+        s.connect(("8.8.8.8", 80))
+        ip = s.getsockname()[0]
+    except Exception:
+        ip = "127.0.0.1"
+    finally:
+        s.close()
+
+    return ip
+
+
+def patch_uuid_encoder():
+    global _IS_JSON_ENCODER_PATCHED
+
+    if not _IS_JSON_ENCODER_PATCHED:
+        JSONEncoder.default = UUIDEncoder.default
+        _IS_JSON_ENCODER_PATCHED = True
+
+
+class UUIDEncoder(JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, UUID):
+            return str(obj)
+        if isinstance(obj, datetime):
+            return obj.timestamp()
+        return super().default(obj)

autotrainer/api/zeromq/__init__.py

@@ -0,0 +1 @@
+from .zeromq_api_service import ZeroMQApiService

autotrainer/api/zeromq/zeromq_api_service.py

@@ -0,0 +1,132 @@
+import json
+import logging
+from typing import Optional
+
+import zmq
+import humps
+
+from ..rpc_service import RpcService, RpcOptions, ApiTopic, ApiCommandRequestServiceResponse, ApiCommandRequest, ApiCommand
+from ..util import get_ip4_addr_str, UUIDEncoder
+
+logger = logging.getLogger(__name__)
+
+
+class ZeroMQApiService(RpcService):
+    def __init__(self, options: RpcOptions):
+        super().__init__(options)
+
+        ips = [get_ip4_addr_str()]
+
+        if ips[0] != "127.0.0.1":
+            ips.append("127.0.0.1")
+
+        self._pub_addresses = [f"tcp://{ip}:{self.subscriber_port}" for ip in ips]
+        self._pub_socket = None
+
+        self._cmd_addresses = [f"tcp://{ip}:{self.command_port}" for ip in ips]
+        self._cmd_socket = None
+
+        self._response_pending = False
+
+    def _start(self) -> bool:
+        if self._pub_socket is not None:
+            return True
+
+        try:
+            context = zmq.Context()
+
+            self._pub_socket = context.socket(zmq.PUB)
+            for address in self._pub_addresses:
+                self._pub_socket.bind(address)
+                logger.debug(f"ZMQ PUB socket bound to {address}")
+
+            context = zmq.Context()
+
+            self._cmd_socket = context.socket(zmq.REP)
+            for address in self._cmd_addresses:
+                self._cmd_socket.bind(address)
+                logger.debug(f"ZMQ REP socket bound to {address}")
+        except zmq.error.ZMQError:
+            self._pub_socket = None
+            self._cmd_socket = None
+            logger.info(f"ZMQ not started.  Address may already be in use.")
+            return False
+
+        return True
+
+    def _stop(self):
+        if self._pub_socket is not None:
+            for address in self._pub_addresses:
+                self._pub_socket.disconnect(address)
+            self._pub_socket = None
+        if self._cmd_socket is not None:
+            for address in self._cmd_addresses:
+                self._cmd_socket.disconnect(address)
+            self._cmd_socket = None
+
+    def _send(self, topic: ApiTopic, data: bytes):
+        if self._pub_socket is not None:
+            self._pub_socket.send(topic.to_bytes(4, "little"), flags=zmq.SNDMORE)
+            self._pub_socket.send(data)
+
+    def _send_string(self, topic: ApiTopic, message: str):
+        if self._pub_socket is not None:
+            self._pub_socket.send(topic.to_bytes(4, "little"), flags=zmq.SNDMORE)
+            self._pub_socket.send(message.encode("utf8"))
+
+    def _send_dict(self, topic: ApiTopic, message: dict):
+        if self._pub_socket is not None:
+            try:
+                # Convert the dictionary to a JSON string and then encode it to bytes.
+                json_data = humps.camelize(json.dumps(message, cls=UUIDEncoder))
+                self._pub_socket.send(topic.to_bytes(4, "little"), flags=zmq.SNDMORE)
+                self._pub_socket.send_json(json_data)
+            except Exception as ex:
+                logger.error(ex)
+
+    def _get_next_command_request(self) -> Optional[ApiCommandRequest]:
+        if self._cmd_socket is not None and not self._response_pending:
+            try:
+                message = self._cmd_socket.recv(flags=zmq.NOBLOCK)
+
+                self._response_pending = True
+
+                request = ZeroMQApiService._parse_command_request(message)
+
+                logger.info(f"Received command request: {request.command}")
+
+                if request is None:
+                    # If a request was received, but could not be parsed, the requester is still expecting a response.
+                    # Otherwise, the ZeroMQ socket on both ends will be in a lingering state.  Send it ourselves since
+                    # returning None will not generate a response by the caller.
+                    self._send_command_response(ApiCommandRequestServiceResponse(command=ApiCommand.NONE, nonce=0))
+
+                # A response will be sent by the caller if a request is returned.
+                return request
+
+            except zmq.Again:
+                # No messages available from recv().
+                pass
+
+        return None
+
+    def _send_command_response(self, response: ApiCommandRequestServiceResponse):
+        if self._cmd_socket is not None:
+            # This is guaranteed to return a valid response, even if modified due to any errors in serialization.
+            data = response.as_bytes(True)
+
+            self._cmd_socket.send(data)
+
+            self._response_pending = False
+
+    @staticmethod
+    def _parse_command_request(message: bytes) -> Optional[ApiCommandRequest]:
+        try:
+            return ApiCommandRequest.parse_bytes(message)
+        except json.decoder.JSONDecodeError as ex:
+            # Might do something different here.
+            logger.error(ex)
+        except Exception as ex:
+            logger.error(ex)
+
+        return None