plastron-stomp 4.3.2__tar.gz

plastron_stomp-4.3.2/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.1
+Name: plastron-stomp
+Version: 4.3.2
+Summary: Plastron STOMP daemon
+Author-email: University of Maryland Libraries <lib-ssdr@umd.edu>, Josh Westgard <westgard@umd.edu>, Peter Eichman <peichman@umd.edu>, Mohamed Abdul Rasheed <mohideen@umd.edu>, Ben Wallberg <wallberg@umd.edu>, David Steelman <dsteelma@umd.edu>, Marc Andreu Grillo Aguilar <aguilarm@umd.edu>
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: BeautifulSoup4
+Requires-Dist: plastron-models
+Requires-Dist: plastron-repo
+Requires-Dist: plastron-utils
+Requires-Dist: pyparsing
+Requires-Dist: PyYAML
+Requires-Dist: stomp.py
+Requires-Dist: watchdog==0.10.3
+Provides-Extra: test
+Requires-Dist: CoilMQ; extra == "test"
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+
+# plastron-stomp
+
+STOMP listener client for asynchronous and synchronous operations
+
+## Running with Python
+
+As a Python module:
+
+```bash
+python -m plastron.stomp.daemon -c <config file>
+```
+
+Using the console script entrypoint:
+
+```bash
+plastrond-stomp -c <config file>
+```
+
+## Integration Tests
+
+See the [integration test README](integration-tests/README.md) for
+instructions on running the manual integration tests.
+
+## Docker Image
+
+The plastron-stomp package contains a [Dockerfile](Dockerfile) for 
+building the `plastrond-stomp` Docker image.
+
+### Building
+
+**Important:** This image **MUST** be built from the main _plastron_ 
+project directory, in order to include the other plastron packages in the 
+build context.
+
+```bash
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+```
+
+### Running with Docker Swarm
+
+This repository contains a [compose.yml](compose.yml) file that defines 
+part of a `plastrond` Docker stack intended to be run alongside the 
+[umd-fcrepo-docker] stack. This repository's configuration adds a 
+`plastrond-stomp` container.
+
+```bash
+# if you are not already running in swarm mode
+docker swarm init
+
+# build the image
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+
+# Create an "archelon_id" private/public key pair
+# The Archelon instance should be configured with "archelon_id.pub" as the
+# PLASTRON_PUBLIC_KEY
+ssh-keygen -q -t rsa -N '' -f archelon_id
+
+# Copy the docker-plastron-template.yml and edit the configuration
+cp docker-plastron.template.yml docker-plastron.yml
+vim docker-plastron.yml
+
+# deploy the stack to run the STOMP application
+docker stack deploy -c plastron-stomp/compose.yml plastrond
+```
+
+To watch the logs:
+
+```bash
+docker service logs -f plastrond_stomp
+```
+
+To stop the STOMP service:
+
+```bash
+docker service rm plastrond_stomp
+```
+
+## Configuration
+
+See [docker-plastron.template.yml](../docker-plastron.template.yml) for an
+example of the config file.
+
+## STOMP Message Headers
+
+The Plastron Daemon expects the following headers to be present in messages
+received on the `JOBS` destination:
+
+* `PlastronCommand`
+* `PlastronJobId`
+
+Additional arguments for a command are sent in headers with the form `PlastronArg-{name}`.
+Many of these are specific to the command, but there is one with standard behavior across
+all commands:
+
+| Header                     | Description                                   |
+|----------------------------|-----------------------------------------------|
+| `PlastronArg-on-behalf-of` | Username to delegate repository operations to |
+
+See the [messages documentation](docs/messages.md) for details on the headers 
+and bodies of the messages the Plastron STOMP Daemon emits.
+
+[umd-fcrepo-docker]: https://github.com/umd-lib/umd-fcrepo-docker

plastron_stomp-4.3.2/README.md

@@ -0,0 +1,104 @@
+# plastron-stomp
+
+STOMP listener client for asynchronous and synchronous operations
+
+## Running with Python
+
+As a Python module:
+
+```bash
+python -m plastron.stomp.daemon -c <config file>
+```
+
+Using the console script entrypoint:
+
+```bash
+plastrond-stomp -c <config file>
+```
+
+## Integration Tests
+
+See the [integration test README](integration-tests/README.md) for
+instructions on running the manual integration tests.
+
+## Docker Image
+
+The plastron-stomp package contains a [Dockerfile](Dockerfile) for 
+building the `plastrond-stomp` Docker image.
+
+### Building
+
+**Important:** This image **MUST** be built from the main _plastron_ 
+project directory, in order to include the other plastron packages in the 
+build context.
+
+```bash
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+```
+
+### Running with Docker Swarm
+
+This repository contains a [compose.yml](compose.yml) file that defines 
+part of a `plastrond` Docker stack intended to be run alongside the 
+[umd-fcrepo-docker] stack. This repository's configuration adds a 
+`plastrond-stomp` container.
+
+```bash
+# if you are not already running in swarm mode
+docker swarm init
+
+# build the image
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+
+# Create an "archelon_id" private/public key pair
+# The Archelon instance should be configured with "archelon_id.pub" as the
+# PLASTRON_PUBLIC_KEY
+ssh-keygen -q -t rsa -N '' -f archelon_id
+
+# Copy the docker-plastron-template.yml and edit the configuration
+cp docker-plastron.template.yml docker-plastron.yml
+vim docker-plastron.yml
+
+# deploy the stack to run the STOMP application
+docker stack deploy -c plastron-stomp/compose.yml plastrond
+```
+
+To watch the logs:
+
+```bash
+docker service logs -f plastrond_stomp
+```
+
+To stop the STOMP service:
+
+```bash
+docker service rm plastrond_stomp
+```
+
+## Configuration
+
+See [docker-plastron.template.yml](../docker-plastron.template.yml) for an
+example of the config file.
+
+## STOMP Message Headers
+
+The Plastron Daemon expects the following headers to be present in messages
+received on the `JOBS` destination:
+
+* `PlastronCommand`
+* `PlastronJobId`
+
+Additional arguments for a command are sent in headers with the form `PlastronArg-{name}`.
+Many of these are specific to the command, but there is one with standard behavior across
+all commands:
+
+| Header                     | Description                                   |
+|----------------------------|-----------------------------------------------|
+| `PlastronArg-on-behalf-of` | Username to delegate repository operations to |
+
+See the [messages documentation](docs/messages.md) for details on the headers 
+and bodies of the messages the Plastron STOMP Daemon emits.
+
+[umd-fcrepo-docker]: https://github.com/umd-lib/umd-fcrepo-docker

plastron_stomp-4.3.2/VERSION

@@ -0,0 +1 @@
+4.3.2

plastron_stomp-4.3.2/pyproject.toml

@@ -0,0 +1,46 @@
+[project]
+name = "plastron-stomp"
+description = "Plastron STOMP daemon"
+authors = [
+    { name='University of Maryland Libraries', email='lib-ssdr@umd.edu' },
+    { name='Josh Westgard', email='westgard@umd.edu' },
+    { name='Peter Eichman', email='peichman@umd.edu' },
+    { name='Mohamed Abdul Rasheed', email='mohideen@umd.edu' },
+    { name='Ben Wallberg', email='wallberg@umd.edu' },
+    { name='David Steelman', email='dsteelma@umd.edu' },
+    { name='Marc Andreu Grillo Aguilar', email='aguilarm@umd.edu' },
+]
+readme = "README.md"
+requires-python = ">= 3.8"
+dependencies = [
+    "click",
+    "BeautifulSoup4",
+    "plastron-models",
+    "plastron-repo",
+    "plastron-utils",
+    "pyparsing",
+    "PyYAML",
+    "stomp.py",
+    'watchdog==0.10.3',
+]
+dynamic = ["version"]
+
+[project.optional-dependencies]
+test = [
+    'CoilMQ',
+    "pytest",
+    "pytest-cov",
+]
+
+[project.scripts]
+plastrond-stomp = 'plastron.stomp.daemon:main'
+
+[build-system]
+requires = ["setuptools>=66.1.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.dynamic]
+version = { "file" = "VERSION" }
+
+[tool.pytest.ini_options]
+markers = ['jobs_dir']

plastron_stomp-4.3.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

plastron_stomp-4.3.2/src/plastron/stomp/__init__.py

@@ -0,0 +1,4 @@
+""".. include:: ../../../README.md"""
+import importlib.metadata
+
+__version__ = importlib.metadata.version('plastron-stomp')

plastron_stomp-4.3.2/src/plastron/stomp/commands/__init__.py

@@ -0,0 +1,42 @@
+from importlib import import_module
+from types import ModuleType
+from typing import Optional, Dict, TypeVar, Type
+
+from plastron.repo import Repository
+
+
+class BaseCommand:
+    def __init__(self, config=None):
+        if config is None:
+            config = {}
+        self.config = config
+        self.repo: Optional[Repository] = None
+        self.result: Optional[Dict] = None
+
+
+T = TypeVar('T', bound=BaseCommand)
+
+
+def get_module_name(command_name: str) -> str:
+    if command_name == 'import':
+        # special case for the import command, to avoid conflict
+        # with the "import" keyword
+        return command_name + 'command'
+    else:
+        return command_name
+
+
+def get_command_module(command_name: str) -> ModuleType:
+    try:
+        return import_module(".".join((__package__, get_module_name(command_name))))
+    except ModuleNotFoundError as e:
+        raise RuntimeError(f'Unable to load a command with the name "{command_name}"') from e
+
+
+def get_command_class(command_name: str) -> Type[BaseCommand]:
+    command_module = get_command_module(command_name)
+    command_class = getattr(command_module, 'Command')
+    if command_class is None:
+        raise RuntimeError(f'Command class not found in module "{command_module}"')
+
+    return command_class

plastron_stomp-4.3.2/src/plastron/stomp/commands/echo.py

@@ -0,0 +1,26 @@
+import logging
+import time
+from typing import Generator, Any, Dict
+
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.repo import Repository
+
+logger = logging.getLogger(__name__)
+
+
+def echo(
+        _repo: Repository,
+        _config: Dict[str, Any],
+        message: PlastronCommandMessage,
+) -> Generator[Any, None, Dict[str, Any]]:
+    message_body = message.body.encode('utf-8').decode('utf-8-sig')
+    echo_delay = int(message.args.get('echo-delay', "0"))
+    if echo_delay:
+        time.sleep(echo_delay)
+
+    yield {'echo': message_body}
+
+    return {
+        'type': 'Done',
+        'echo': message_body,
+    }

plastron_stomp-4.3.2/src/plastron/stomp/commands/export.py

@@ -0,0 +1,28 @@
+import logging
+from typing import Generator, Any, Dict
+
+from plastron.context import PlastronContext
+from plastron.jobs.exportjob import ExportJob
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.utils import strtobool
+
+logger = logging.getLogger(__name__)
+
+
+def export(
+        context: PlastronContext,
+        message: PlastronCommandMessage,
+) -> Generator[Dict[str, Any], None, Dict[str, Any]]:
+    ssh_key = context.config.get('COMMANDS', {}).get('EXPORT', {}).get('SSH_PRIVATE_KEY', None)
+    export_job = ExportJob(
+        context=context,
+        export_binaries=bool(strtobool(message.args.get('export-binaries', 'false'))),
+        binary_types=message.args.get('binary-types'),
+        uris=message.body.strip().split('\n'),
+        export_format=message.args.get('format', 'text/turtle'),
+        output_dest=message.args.get('output-dest'),
+        uri_template=message.args.get('uri-template'),
+        key=ssh_key,
+    )
+    logger.info(f'Received message to initiate export job {message.job_id} containing {len(export_job.uris)} items')
+    return export_job.run()

plastron_stomp-4.3.2/src/plastron/stomp/commands/importcommand.py

@@ -0,0 +1,84 @@
+import io
+import logging
+from argparse import ArgumentTypeError
+from typing import Any, Dict, Generator, Optional
+
+from rdflib import URIRef
+
+from plastron.context import PlastronContext
+from plastron.jobs import Jobs
+from plastron.jobs.importjob import ImportConfig, ImportJob
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.utils import datetimestamp, strtobool, uri_or_curie
+
+logger = logging.getLogger(__name__)
+
+
+def get_access_uri(access) -> Optional[URIRef]:
+    if access is None:
+        return None
+    try:
+        return uri_or_curie(access)
+    except ArgumentTypeError as e:
+        raise RuntimeError(f'PlastronArg-access {e}')
+
+
+def importcommand(
+        context: PlastronContext,
+        message: PlastronCommandMessage,
+) -> Generator[Dict[str, Any], None, Dict[str, Any]]:
+    """
+    Performs the import
+
+    :param context
+    :param message:
+    """
+    job_id = message.job_id
+
+    # per-request options that are NOT saved to the config
+    limit = message.args.get('limit', None)
+    if limit is not None:
+        limit = int(limit)
+    message.body = message.body.encode('utf-8').decode('utf-8-sig')
+    percentage = message.args.get('percent', None)
+    validate_only = bool(strtobool(message.args.get('validate-only', 'false')))
+    publish = bool(strtobool(message.args.get('publish', 'false')))
+    resume = bool(strtobool(message.args.get('resume', 'false')))
+    import_file = io.StringIO(message.body)
+
+    # options that are saved to the config
+    job_config_args = {
+        'job_id': job_id,
+        'model': message.args.get('model'),
+        'access': get_access_uri(message.args.get('access')),
+        'member_of': message.args.get('member-of'),
+        'container': message.args.get('relpath'),
+        'binaries_location': message.args.get('binaries-location'),
+    }
+
+    if resume and job_id is None:
+        raise RuntimeError('Resuming a job requires a job id')
+
+    if job_id is None:
+        # TODO: generate a more unique id? add in user and hostname?
+        job_id = f"import-{datetimestamp()}"
+
+    config = context.config.get('COMMANDS', {}).get('IMPORT', {})
+    jobs = Jobs(directory=config.get('JOBS_DIR', 'jobs'))
+    if resume:
+        job = jobs.get_job(ImportJob, job_id=job_id)
+        # update the config with any changes in this request
+        job.update_config(job_config_args)
+    else:
+        job = jobs.create_job(ImportJob, config=ImportConfig(**job_config_args))
+
+    job.ssh_private_key = config.get('SSH_PRIVATE_KEY', None)
+
+    return job.run(
+        context=context,
+        import_file=import_file,
+        limit=limit,
+        percentage=percentage,
+        validate_only=validate_only,
+        publish=publish,
+    )

plastron_stomp-4.3.2/src/plastron/stomp/commands/publish.py

@@ -0,0 +1,20 @@
+from typing import Dict, Any, Generator
+
+from plastron.context import PlastronContext
+from plastron.jobs.publicationjob import PublicationJob, PublicationAction
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.utils import strtobool
+
+
+def publish(
+        context: PlastronContext,
+        message: PlastronCommandMessage,
+) -> Generator[Dict[str, Any], None, Dict[str, Any]]:
+    job = PublicationJob(
+        context=context,
+        uris=message.body.strip().split('\n'),
+        action=PublicationAction.PUBLISH,
+        force_hidden=bool(strtobool(message.args.get('hidden', 'false'))),
+        force_visible=bool(strtobool(message.args.get('visible', 'false'))),
+    )
+    return job.run()

plastron_stomp-4.3.2/src/plastron/stomp/commands/unpublish.py

@@ -0,0 +1,20 @@
+from typing import Dict, Any, Generator
+
+from plastron.context import PlastronContext
+from plastron.jobs.publicationjob import PublicationJob, PublicationAction
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.utils import strtobool
+
+
+def unpublish(
+        context: PlastronContext,
+        message: PlastronCommandMessage,
+) -> Generator[Dict[str, Any], None, Dict[str, Any]]:
+    job = PublicationJob(
+        context=context,
+        uris=message.body.strip().split('\n'),
+        action=PublicationAction.UNPUBLISH,
+        force_hidden=bool(strtobool(message.args.get('hidden', 'false'))),
+        force_visible=bool(strtobool(message.args.get('visible', 'false'))),
+    )
+    return job.run()

plastron_stomp-4.3.2/src/plastron/stomp/commands/update.py

@@ -0,0 +1,45 @@
+import json
+import logging
+from typing import Generator, Any, Dict
+
+from plastron.context import PlastronContext
+from plastron.jobs.updatejob import UpdateJob
+from plastron.messaging.messages import PlastronCommandMessage
+from plastron.models import get_model_class
+from plastron.utils import strtobool, parse_predicate_list
+
+logger = logging.getLogger(__name__)
+
+
+def parse_message(message: PlastronCommandMessage) -> Dict[str, Any]:
+    message.body = message.body.encode('utf-8').decode('utf-8-sig')
+    body = json.loads(message.body)
+    uris = body['uris']
+    sparql_update = body['sparql_update']
+    validate = bool(strtobool(message.args.get('validate', 'false')))
+    model = message.args.get('model', None)
+    recursive = message.args.get('recursive', None)
+
+    if validate and not model:
+        raise RuntimeError("Model must be provided when performing validation")
+
+    # Retrieve the model to use for validation
+    model_class = get_model_class(model) if model else None
+
+    traverse = parse_predicate_list(recursive) if recursive is not None else []
+    return {
+        'uris': uris,
+        'sparql_update': sparql_update,
+        'model_class': model_class,
+        'traverse': traverse,
+        'dry_run': bool(strtobool(message.args.get('dry-run', 'false'))),
+        # Default to no transactions, due to LIBFCREPO-842
+        'use_transactions': not bool(strtobool(message.args.get('no-transactions', 'true'))),
+    }
+
+
+def update(
+        context: PlastronContext,
+        message: PlastronCommandMessage,
+) -> Generator[Dict[str, str], None, Dict[str, Any]]:
+    return UpdateJob(repo=context.repo, **parse_message(message)).run()

plastron_stomp-4.3.2/src/plastron/stomp/daemon.py

@@ -0,0 +1,116 @@
+import logging
+import logging.config
+import os
+import sys
+from datetime import datetime
+from pathlib import Path
+from threading import Event, Thread
+from typing import TextIO, Dict, Any
+
+import click
+import yaml
+
+from plastron.context import PlastronContext
+from plastron.stomp import __version__
+from plastron.stomp.listeners import CommandListener
+from plastron.utils import DEFAULT_LOGGING_OPTIONS, envsubst
+
+logger = logging.getLogger(__name__)
+
+
+class STOMPDaemon(Thread):
+    def __init__(self, config: Dict[str, Any], **kwargs):
+        super().__init__(**kwargs)
+        self.context = PlastronContext(config)
+        self.started = Event()
+        self.stopped = Event()
+        self.broker = self.context.broker
+
+        def started():
+            self.stopped.clear()
+            self.started.set()
+
+        def stopped():
+            self.started.clear()
+            self.stopped.set()
+
+        self.command_listener = CommandListener(
+            context=self.context,
+            after_connected=started,
+            after_disconnected=stopped,
+        )
+
+    def run(self):
+        # setup listeners
+        self.broker.set_listener('command', self.command_listener)
+
+        # connect and listen until the stopped Event is set
+        if self.broker.connect(client_id=f'plastrond/{__version__}-{os.uname().nodename}-{os.getpid()}'):
+            self.stopped.wait()
+
+            self.broker.disconnect()
+            if self.command_listener.inbox_watcher:
+                self.command_listener.inbox_watcher.stop()
+        else:
+            logger.error('Unable to connect to STOMP broker')
+
+
+def configure_logging(log_filename_base: str, log_dir: str = 'logs', verbose: bool = False) -> None:
+    logging_options = DEFAULT_LOGGING_OPTIONS
+
+    # log file configuration
+    log_dirname = Path(log_dir)
+    log_dirname.mkdir(parents=True, exist_ok=True)
+    now = datetime.utcnow().strftime('%Y%m%d%H%M%S')
+    log_filename = '.'.join((log_filename_base, now, 'log'))
+    logfile = log_dirname / log_filename
+    logging_options['handlers']['file']['filename'] = str(logfile)
+
+    # manipulate console verbosity
+    if verbose:
+        logging_options['handlers']['console']['level'] = 'DEBUG'
+
+    # configure logging
+    logging.config.dictConfig(logging_options)
+
+
+@click.command
+@click.option(
+    '-c', '--config-file',
+    type=click.File(),
+    help='Configuration file',
+    required=True,
+)
+@click.option(
+    '-v', '--verbose',
+    is_flag=True,
+    help='increase the verbosity of the status output',
+)
+def main(config_file: TextIO, verbose: bool):
+    config = envsubst(yaml.safe_load(config_file))
+    repo_config = config.get('REPOSITORY', {})
+    configure_logging(
+        log_filename_base='plastron.daemon',
+        log_dir=repo_config.get('LOG_DIR', 'logs'),
+        verbose=verbose,
+    )
+    daemon_description = f'plastrond-stomp/{__version__}'
+    logger.info(f'Starting {daemon_description}')
+    thread = STOMPDaemon(config=config)
+
+    try:
+        thread.start()
+        while thread.is_alive():
+            thread.join(1)
+    except KeyboardInterrupt:
+        logger.warning(f'Shutting down {daemon_description}')
+        if hasattr(thread, 'stopped'):
+            thread.stopped.set()
+            thread.stopped.wait()
+
+    logger.info(f'{daemon_description} shut down successfully')
+    sys.exit()
+
+
+if __name__ == "__main__":
+    main()

plastron_stomp-4.3.2/src/plastron/stomp/handlers.py

@@ -0,0 +1,55 @@
+import logging
+
+from plastron.messaging.broker import Destination
+from plastron.messaging.messages import PlastronMessage, PlastronErrorMessage
+
+logger = logging.getLogger(__name__)
+
+
+class AsynchronousResponseHandler:
+    def __init__(self, listener, message: PlastronMessage):
+        self.listener = listener
+        self.message = message
+        self.reply_queue: Destination = listener.broker['JOB_STATUS']
+
+    def __call__(self, future):
+        e = future.exception()
+        if e:
+            logger.error(f"Job {self.message.job_id} failed: {e}")
+            response = PlastronErrorMessage(job_id=self.message.job_id, error=str(e))
+        else:
+            # assume no errors, return the response
+            response = future.result()
+
+        # save a copy of the response message in the outbox
+        job_id = response.job_id
+        self.listener.outbox.add(job_id, response)
+
+        # remove the message from the inbox now that processing has completed
+        self.listener.inbox.remove(self.message.id)
+
+        # send the job completed message
+        self.reply_queue.send(response)
+        logger.debug(f'Response message sent to {self.reply_queue} with headers: {response.headers}')
+
+        # remove the message from the outbox now that sending has completed
+        self.listener.outbox.remove(job_id)
+
+
+class SynchronousResponseHandler:
+    def __init__(self, listener, message: PlastronMessage):
+        self.listener = listener
+        self.message = message
+        self.reply_queue = Destination(self.listener.broker, message.headers['reply-to'])
+
+    def __call__(self, future):
+        e = future.exception()
+        if e:
+            logger.error(f"Job {self.message.job_id} failed: {e}")
+            self.reply_queue.send(PlastronErrorMessage(job_id=self.message.job_id, error=str(e)))
+        else:
+            # assume no errors, return the response
+            response = future.result()
+            # send to the specified "reply to" queue
+            self.reply_queue.send(response)
+            logger.debug(f'Response message sent to {self.reply_queue} with headers: {response.headers}')

plastron_stomp-4.3.2/src/plastron/stomp/inbox_watcher.py

@@ -0,0 +1,49 @@
+import logging
+
+from plastron.stomp.handlers import AsynchronousResponseHandler
+from watchdog.observers import Observer
+from watchdog.events import FileSystemEventHandler, FileCreatedEvent
+
+
+logger = logging.getLogger(__name__)
+
+
+class InboxEventHandler(FileSystemEventHandler):
+    """Triggers message processing when a file is added in the
+       inbox directory."""
+    # Note to maintainers: The original implementation of this class included
+    # both "on_created" and "on_modified" event handlers. On Mac OS X, new file
+    # creation only triggers the "on_created" event. On Linux, new file
+    # creation triggers both an "on_created" event and "on_modified" event,
+    # leading to duplicate processing (see LIBFCREPO-821).
+    def __init__(self, command_listener, message_box):
+        self.command_listener = command_listener
+        self.message_box = message_box
+
+    def on_created(self, event):
+        if isinstance(event, FileCreatedEvent):
+            logger.info(f"Triggering inbox processing due to {event}")
+            message = self.message_box.message_class.read(event.src_path)
+            self.command_listener.process_message(message, AsynchronousResponseHandler(self.command_listener, message))
+
+
+class InboxWatcher:
+    """
+    Watches for changes to the inbox directory, in order to trigger message
+    processing via InboxEventHandler
+    """
+    def __init__(self, command_listener, message_box):
+        """Constructs the watchdog Observer"""
+        self.observer = Observer()
+        self.observer.schedule(InboxEventHandler(command_listener, message_box), message_box.dir, recursive=True)
+
+    def start(self):
+        """Start the watcher"""
+        logger.debug("Starting InboxWatcher")
+        self.observer.start()
+
+    def stop(self):
+        """Stop the watcher"""
+        logger.debug("Stopping InboxWatcher")
+        self.observer.unschedule_all()
+        self.observer.stop()

plastron_stomp-4.3.2/src/plastron/stomp/listeners.py

@@ -0,0 +1,143 @@
+import importlib.metadata
+import logging
+import os
+from concurrent.futures import ThreadPoolExecutor
+from typing import Dict, Any, Callable, Generator, Iterator
+
+from stomp.listener import ConnectionListener
+
+from plastron.context import PlastronContext
+from plastron.messaging.broker import Destination
+from plastron.messaging.messages import MessageBox, PlastronCommandMessage, PlastronMessage
+from plastron.stomp.commands import get_command_module, get_module_name
+from plastron.stomp.handlers import AsynchronousResponseHandler, SynchronousResponseHandler
+from plastron.stomp.inbox_watcher import InboxWatcher
+
+logger = logging.getLogger(__name__)
+version = importlib.metadata.version('plastron-stomp')
+
+
+class CommandListener(ConnectionListener):
+    def __init__(self, context: PlastronContext, after_connected: Callable = None, after_disconnected: Callable = None):
+        self.context = context
+        self.broker = context.broker
+        self.inbox = MessageBox(os.path.join(self.broker.message_store_dir, 'inbox'), PlastronCommandMessage)
+        self.outbox = MessageBox(os.path.join(self.broker.message_store_dir, 'outbox'), PlastronMessage)
+        self.executor = ThreadPoolExecutor(thread_name_prefix=__name__)
+        self.inbox_watcher = None
+        self.processor = MessageProcessor(context)
+        self.after_connected = after_connected
+        self.after_disconnected = after_disconnected
+
+    def on_connecting(self, host_and_port):
+        logger.info(f'Connecting to STOMP message broker {self.broker}')
+
+    def on_connected(self, frame):
+        logger.info(f'Connected to STOMP message broker {self.broker}')
+
+        # first attempt to send anything in the outbox
+        for message in self.outbox:
+            logger.info(f"Found response message for job {message.job_id} in outbox")
+            # send the job completed message
+            self.broker['JOB_STATUS'].send(message)
+            logger.info(f'Sent response message for job {message.job_id}')
+            # remove the message from the outbox now that sending has completed
+            self.outbox.remove(message.job_id)
+
+        # then process anything in the inbox
+        for message in self.inbox:
+            self.process_message(message, AsynchronousResponseHandler(self, message))
+
+        # subscribe to receive asynchronous jobs
+        self.broker['JOBS'].subscribe(id='plastron', ack='client-individual')
+        # subscribe to receive synchronous jobs
+        self.broker['SYNCHRONOUS_JOBS'].subscribe(id='plastron-synchronous', ack='client-individual')
+
+        self.inbox_watcher = InboxWatcher(self, self.inbox)
+        self.inbox_watcher.start()
+
+        if self.after_connected:
+            self.after_connected()
+
+    def on_message(self, frame):
+        headers = frame.headers
+        body = frame.body
+        logger.debug(f'Received message on {headers["destination"]} with headers: {headers}')
+        if headers['destination'] == self.broker['SYNCHRONOUS_JOBS'].name:
+            message = PlastronCommandMessage(headers=headers, body=body)
+            self.process_message(message, SynchronousResponseHandler(self, message))
+            self.broker.ack(message.id, 'plastron-synchronous')
+
+        elif headers['destination'] == self.broker['JOBS'].name:
+            # save the message in the inbox until we can process it
+            # Note: Processing will occur via the InboxWatcher, which will
+            # respond to the inbox placing a file in the inbox message directory
+            # containing the message
+            message = PlastronCommandMessage(headers=headers, body=body)
+            self.inbox.add(message.id, message)
+            self.broker.ack(message.id, 'plastron')
+
+    def process_message(self, message, response_handler):
+        # send to a message processor thread
+        self.executor.submit(self.processor, message, self.broker['JOB_PROGRESS']).add_done_callback(response_handler)
+
+    def on_disconnected(self):
+        logger.warning('Disconnected from the STOMP message broker')
+        if self.inbox_watcher:
+            self.inbox_watcher.stop()
+        if self.after_disconnected:
+            self.after_disconnected()
+
+
+# type alias for command functions that take a context and a message, and return a generator that yields
+# status updates in the form of dictionaries, and returns a final state, also as a dictionary
+STOMPCommandFunction = Callable[[PlastronContext, PlastronMessage], Generator[Dict[str, Any], None, Dict[str, Any]]]
+
+
+def get_command(command_name: str) -> STOMPCommandFunction:
+    module_name = get_module_name(command_name)
+    module = get_command_module(command_name)
+    command = getattr(module, module_name, None)
+    if command is None:
+        raise RuntimeError(f'Command function "{module_name}" not found in {module.__name__}')
+    return command
+
+
+class MessageProcessor:
+    def __init__(self, context: PlastronContext):
+        self.context = context
+        # cache for command instances
+        self.commands = {}
+        self.result = None
+
+    def __call__(self, message: PlastronCommandMessage, progress_topic: Destination):
+        if message.job_id is None:
+            raise RuntimeError('Expecting a PlastronJobId header')
+
+        logger.info(f'Received message to initiate job {message.job_id}')
+
+        # determine which command to load to process the message
+        command = get_command(message.command)
+
+        delegated_user = message.args.get('on-behalf-of')
+        if delegated_user is not None:
+            logger.info(f'Running repository operations on behalf of {delegated_user}')
+
+        # run the command, and send a progress message over STOMP every time it yields
+        # the _run() delegating generator captures the final status in self.result
+        with self.context.repo_configuration(
+            delegated_user=delegated_user,
+            ua_string=f'plastron/{version}',
+        ) as run_context:
+            for status in self._run(command(run_context, message)):
+                progress_topic.send(PlastronMessage(job_id=message.job_id, body=status))
+
+        logger.info(f'Job {message.job_id} complete')
+
+        # default message state is "Done"
+        return message.response(state=self.result.get('type', 'Done'), body=self.result)
+
+    def _run(self, command: Generator[Dict, None, Dict]) -> Iterator[Dict[str, Any]]:
+        # delegating generator; each progress step is passed to the calling
+        # method, and the return value from the command is stored as the result
+        self.result = yield from command

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.1
+Name: plastron-stomp
+Version: 4.3.2
+Summary: Plastron STOMP daemon
+Author-email: University of Maryland Libraries <lib-ssdr@umd.edu>, Josh Westgard <westgard@umd.edu>, Peter Eichman <peichman@umd.edu>, Mohamed Abdul Rasheed <mohideen@umd.edu>, Ben Wallberg <wallberg@umd.edu>, David Steelman <dsteelma@umd.edu>, Marc Andreu Grillo Aguilar <aguilarm@umd.edu>
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: BeautifulSoup4
+Requires-Dist: plastron-models
+Requires-Dist: plastron-repo
+Requires-Dist: plastron-utils
+Requires-Dist: pyparsing
+Requires-Dist: PyYAML
+Requires-Dist: stomp.py
+Requires-Dist: watchdog==0.10.3
+Provides-Extra: test
+Requires-Dist: CoilMQ; extra == "test"
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+
+# plastron-stomp
+
+STOMP listener client for asynchronous and synchronous operations
+
+## Running with Python
+
+As a Python module:
+
+```bash
+python -m plastron.stomp.daemon -c <config file>
+```
+
+Using the console script entrypoint:
+
+```bash
+plastrond-stomp -c <config file>
+```
+
+## Integration Tests
+
+See the [integration test README](integration-tests/README.md) for
+instructions on running the manual integration tests.
+
+## Docker Image
+
+The plastron-stomp package contains a [Dockerfile](Dockerfile) for 
+building the `plastrond-stomp` Docker image.
+
+### Building
+
+**Important:** This image **MUST** be built from the main _plastron_ 
+project directory, in order to include the other plastron packages in the 
+build context.
+
+```bash
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+```
+
+### Running with Docker Swarm
+
+This repository contains a [compose.yml](compose.yml) file that defines 
+part of a `plastrond` Docker stack intended to be run alongside the 
+[umd-fcrepo-docker] stack. This repository's configuration adds a 
+`plastrond-stomp` container.
+
+```bash
+# if you are not already running in swarm mode
+docker swarm init
+
+# build the image
+docker build -t docker.lib.umd.edu/plastrond-stomp:latest \
+    -f plastron-stomp/Dockerfile .
+
+# Create an "archelon_id" private/public key pair
+# The Archelon instance should be configured with "archelon_id.pub" as the
+# PLASTRON_PUBLIC_KEY
+ssh-keygen -q -t rsa -N '' -f archelon_id
+
+# Copy the docker-plastron-template.yml and edit the configuration
+cp docker-plastron.template.yml docker-plastron.yml
+vim docker-plastron.yml
+
+# deploy the stack to run the STOMP application
+docker stack deploy -c plastron-stomp/compose.yml plastrond
+```
+
+To watch the logs:
+
+```bash
+docker service logs -f plastrond_stomp
+```
+
+To stop the STOMP service:
+
+```bash
+docker service rm plastrond_stomp
+```
+
+## Configuration
+
+See [docker-plastron.template.yml](../docker-plastron.template.yml) for an
+example of the config file.
+
+## STOMP Message Headers
+
+The Plastron Daemon expects the following headers to be present in messages
+received on the `JOBS` destination:
+
+* `PlastronCommand`
+* `PlastronJobId`
+
+Additional arguments for a command are sent in headers with the form `PlastronArg-{name}`.
+Many of these are specific to the command, but there is one with standard behavior across
+all commands:
+
+| Header                     | Description                                   |
+|----------------------------|-----------------------------------------------|
+| `PlastronArg-on-behalf-of` | Username to delegate repository operations to |
+
+See the [messages documentation](docs/messages.md) for details on the headers 
+and bodies of the messages the Plastron STOMP Daemon emits.
+
+[umd-fcrepo-docker]: https://github.com/umd-lib/umd-fcrepo-docker

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/SOURCES.txt

@@ -0,0 +1,24 @@
+README.md
+VERSION
+pyproject.toml
+src/plastron/stomp/__init__.py
+src/plastron/stomp/daemon.py
+src/plastron/stomp/handlers.py
+src/plastron/stomp/inbox_watcher.py
+src/plastron/stomp/listeners.py
+src/plastron/stomp/commands/__init__.py
+src/plastron/stomp/commands/echo.py
+src/plastron/stomp/commands/export.py
+src/plastron/stomp/commands/importcommand.py
+src/plastron/stomp/commands/publish.py
+src/plastron/stomp/commands/unpublish.py
+src/plastron/stomp/commands/update.py
+src/plastron_stomp.egg-info/PKG-INFO
+src/plastron_stomp.egg-info/SOURCES.txt
+src/plastron_stomp.egg-info/dependency_links.txt
+src/plastron_stomp.egg-info/entry_points.txt
+src/plastron_stomp.egg-info/requires.txt
+src/plastron_stomp.egg-info/top_level.txt
+tests/test_connection.py
+tests/test_handlers.py
+tests/test_inbox_watcher.py

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+plastrond-stomp = plastron.stomp.daemon:main

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/requires.txt

@@ -0,0 +1,14 @@
+click
+BeautifulSoup4
+plastron-models
+plastron-repo
+plastron-utils
+pyparsing
+PyYAML
+stomp.py
+watchdog==0.10.3
+
+[test]
+CoilMQ
+pytest
+pytest-cov

plastron_stomp-4.3.2/src/plastron_stomp.egg-info/top_level.txt

@@ -0,0 +1 @@
+plastron

plastron_stomp-4.3.2/tests/test_connection.py

@@ -0,0 +1,76 @@
+from threading import Thread
+
+import pytest
+from coilmq.protocol import STOMP11
+from coilmq.queue import QueueManager
+from coilmq.server.socket_server import StompServer
+from coilmq.store.memory import MemoryQueue
+from coilmq.topic import TopicManager
+
+from plastron.messaging.broker import ServerTuple, Broker
+from plastron.stomp.daemon import STOMPDaemon
+
+
+@pytest.fixture()
+def server_address() -> ServerTuple:
+    return ServerTuple('localhost', 61613)
+
+
+@pytest.fixture()
+def broker(server_address, shared_datadir) -> Broker:
+    return Broker(
+        server=server_address,
+        message_store_dir=(shared_datadir / 'msg'),
+        destinations={
+            'JOBS': '/queue/plastron.jobs',
+            'JOB_PROGRESS': '/topic/plastron.jobs.progress',
+            'JOB_STATUS': '/queue/plastron.jobs.status',
+            'SYNCHRONOUS_JOBS': '/queue/plastron.jobs.synchronous',
+            'REINDEXING': '/queue/reindex',
+        }
+    )
+
+
+@pytest.fixture()
+def stomp_server(server_address):
+    server = StompServer(
+        server_address=server_address,
+        queue_manager=QueueManager(store=MemoryQueue()),
+        topic_manager=TopicManager(),
+        protocol=STOMP11,
+    )
+
+    yield server
+
+    server.server_close()
+
+
+@pytest.fixture()
+def plastrond_stomp(broker):
+    plastrond = STOMPDaemon(broker=broker, repo_config={})
+
+    yield plastrond
+
+    if plastrond.is_alive():
+        plastrond.stopped.set()
+        plastrond.stopped.wait()
+
+
+@pytest.mark.skip('intermittent failures, test is unreliable')
+def test_stomp_server_closed(stomp_server, plastrond_stomp):
+    server_thread = Thread(target=stomp_server.serve_forever)
+    server_thread.start()
+    assert server_thread.is_alive()
+
+    plastrond_stomp.start()
+    assert plastrond_stomp.is_alive()
+    # wait for startup and subscriptions to happen
+    plastrond_stomp.started.wait()
+
+    # stop the server
+    stomp_server.shutdown()
+
+    # wait for the daemon to shut down
+    plastrond_stomp.stopped.wait(10)
+
+    assert not plastrond_stomp.is_alive()

plastron_stomp-4.3.2/tests/test_handlers.py

@@ -0,0 +1,80 @@
+from concurrent.futures import Future
+from typing import cast, Dict, Type
+from unittest import TestCase
+from unittest.mock import Mock
+
+import pytest
+
+from plastron.messaging.broker import Destination
+from plastron.messaging.messages import MessageBox, PlastronCommandMessage, PlastronErrorMessage, PlastronMessage, \
+    Message
+from plastron.stomp.handlers import AsynchronousResponseHandler
+from plastron.stomp.listeners import CommandListener
+
+
+@pytest.fixture
+def mock_listener():
+    return Mock(
+        spec=CommandListener,
+        inbox=Mock(MessageBox),
+        outbox=Mock(MessageBox),
+        broker={'JOB_STATUS': Mock(Destination)},
+    )
+
+
+def test_asynchronous_response_handler_successful_call_removes_inbox_and_outbox_entries(mock_listener):
+    # Set up mocks
+    job_id = 'test_asynchronous_response_handler_successful_call-123'
+
+    # This future represents a successful call
+    future_response = PlastronMessage(headers={'PlastronJobId': job_id}, body="Success!")
+    future_mock_attrs = {'exception.return_value': None, 'result.return_value': future_response}
+    future = Mock(Future, **future_mock_attrs)
+
+    incoming_message = Mock(PlastronCommandMessage, id='incoming_test_message', job_id=job_id)
+
+    # Handle the incoming message
+    handler = AsynchronousResponseHandler(mock_listener, incoming_message)
+    handler(future)
+
+    # Verify outbox/inbox handling and message sending
+    mock_listener.outbox.add.assert_called_once_with(job_id, future_response)
+    mock_listener.inbox.remove.assert_called_once_with(incoming_message.id)
+    mock_listener.broker['JOB_STATUS'].send.assert_called_once_with(future_response)
+    mock_listener.outbox.remove.assert_called_once_with(job_id)
+
+
+def test_asynchronous_response_handler_call_with_exception_removes_inbox_and_outbox_entries(mock_listener):
+    # Set up mocks
+    job_id = 'test_asynchronous_response_handler_call_with_exception-123'
+
+    # This future throw an exception, representing a failed call
+    future_response = PlastronErrorMessage(headers={'PlastronJobId': job_id})
+    exception_message = 'An error occurred'
+    future_mock_attrs = {'exception.return_value': Exception(exception_message), 'result.return_value': future_response}
+    future = Mock(Future, **future_mock_attrs)
+
+    incoming_message = Mock(PlastronCommandMessage, id='incoming_test_message', job_id=job_id)
+
+    # Handle the incoming message
+    handler = AsynchronousResponseHandler(mock_listener, incoming_message)
+    handler(future)
+
+    # Verify outbox/inbox handling and message sending
+    mock_listener.outbox.add.assert_called_once()
+    mock_listener.inbox.remove.assert_called_once_with(incoming_message.id)
+
+    mock_listener.broker['JOB_STATUS'].send.assert_called_once()
+    expected_msg_headers = {'PlastronJobId': job_id, 'PlastronJobError': exception_message, 'persistent': 'true'}
+    assert_sent_message(mock_listener.broker['JOB_STATUS'], PlastronErrorMessage, expected_msg_headers, '')
+
+    mock_listener.outbox.remove.assert_called_once_with(job_id)
+
+
+def assert_sent_message(queue: Destination, message_class: Type[Message],
+                        expected_message_headers: Dict[str, str], expected_body: str):
+    status_queue_send_args = cast(Mock, queue).send.call_args[0]  # using cast, so mypy doesn't complain
+    sent_message = status_queue_send_args[0]
+    assert isinstance(sent_message, message_class)
+    TestCase().assertDictEqual(sent_message.headers, expected_message_headers)
+    assert sent_message.body == expected_body

plastron_stomp-4.3.2/tests/test_inbox_watcher.py

@@ -0,0 +1,52 @@
+import os
+import tempfile
+import time
+from contextlib import contextmanager
+from unittest.mock import patch
+
+from plastron.stomp.inbox_watcher import InboxWatcher
+from plastron.messaging.messages import MessageBox, PlastronMessage
+
+
+def test_new_file_in_inbox():
+    with tempfile.TemporaryDirectory() as inbox_dirname:
+        with patch('plastron.stomp.listeners.CommandListener') as mock_command_listener:
+            # Mock "process_message" and use to verify that CommandListener
+            # is called (when a file is created in the inbox).
+            mock_method = mock_command_listener.process_message
+
+            with inbox_watcher(inbox_dirname, mock_command_listener):
+                create_test_file(inbox_dirname)
+
+                wait_until_called(mock_method)
+
+                mock_method.assert_called_once()
+
+# Utility methods
+
+
+@contextmanager
+def inbox_watcher(inbox_dirname, mock_command_listener):
+    inbox = MessageBox(inbox_dirname, PlastronMessage)
+    watcher = InboxWatcher(mock_command_listener, inbox)
+    watcher.start()
+
+    try:
+        yield
+    finally:
+        watcher.stop()
+
+
+def create_test_file(inbox_dirname):
+    temp_file = open(os.path.join(inbox_dirname, 'test_new_file'), 'w')
+    temp_file.write("test:test")
+    temp_file.close()
+
+
+def wait_until_called(mock_method, interval=0.1, timeout=5):
+    '''Polls at the given interval until either the mock_method is called
+       or the timeout occurs.'''
+    # Inspired by https://stackoverflow.com/a/36040926
+    start = time.time()
+    while not mock_method.called and time.time() - start < timeout:
+        time.sleep(interval)